My question is about a best practise.
Im sending all simple user page view to redis. What i want to do is for every user,
There should be a list of pages that user ve looked.
That list needs to have a limit of max 20 items.
Lastly, that list needs to have an expiration time (key expiration).
The implemantation is (Im using StackExchange.Redis for applications):
var transaction = _cache.CreateTransaction();
transaction.ListLeftPushAsync(key, JsonConvert.SerializeObject(value), When.Always, CommandFlags.FireAndForget);
transaction.KeyExpireAsync(key, TimeSpan.FromDays(Constants.Constants.TopnUserCacheDurationinDays), CommandFlags.FireAndForget);
if (new Random().Next(7) == 6)
{
transaction.ListTrimAsync(key, 0, Constants.Constants.TopNUserHistoryLimit, CommandFlags.FireAndForget);
}
return transaction.ExecuteAsync();
The question is, is this implemantation is good for my needs. I feel that smth is not right. Because there are ~300 requests in a second. So im sending 1 request with 3 jobs 300 times in a second.
Any suggestions?
Related
There are nearly 14 behaviors of sending points to users, such as user registration, login, purchase, and chat. It is required not to change the previous interface code in Go.
id
motion
points
remark
1
/register
5
one user only have one chance to get points
2
/login
5
every day the user has one chance to get points
3
/comment
3
every day the user has five chance to get points
4
/pay
10
every day the user has three chance to get points
5
/invite
10
every day the user has three chance to get points
6
/send
10
every day the user has three chance to get points
7
/purchase
10
every day the user has three chance to get points
Every day the user only has 100 points
Through the following partial code of /register every user can get 5 points after he registers successfully.
func (r *User) CreateUser(CreateUser *CreateUserModel, c *lmhttp.Context) {
err = r.userDB.insertScoreTx(CreateUser.UserID, 1, 5)
if err != nil {
r.Error("Send points failed", zap.Error(err))
c.ResponseError(errors.New("Send points failed"))
return
}
c.Response(LoginUserDetail(UserModel, token, r.ctx))
}
And the data of table user_points is as below:
id
user_id
points
points_type
1
1
5
1
Can I write a general function of sending points, and then call or check it in a certain place, that is, judge whether to increase points when each interface request is successful, such as the following places where each interface responds successfully?
// Response OK
func (c *Context) ResponseOK() {
c.JSON(http.StatusOK, gin.H{
"status": http.StatusOK,
})
}
Thanks so much for any advice
You can definitely do what you're asking. I don't know Go but some webservers will have a hook to run a function on every successful request. In that function based on the URL you can check if to add points for the user. You are already storing the type of points but you also need to store how many chances they used up for the day.
Then you can write the logic for whether they add points for each of the rules.
If in Go there is no hook function that runs on any request you can go as basic as a function that takes in e.g 'register', 'login'. And call it manually from everywhere you handle the requests you wanna add points for.
You also need to reset the data based on the date since all the rules are "today".
My best advice for that is to also store the current date on all the user_points entries. And when checking whether to add points if the current day is not today you set the number of used up chances to 1 and the date to current.
e.g table
id
user_id
points
points_type
chances_used
date
1
1
5
1
2
14:33 09/02/2023
e.g logic from inside the function:
if type == 'comment':
points = db.get(user_points).filter(user.id=1, points_type=3)
if points.date == date.today():
points.chances_used += 1
if points.chances_used < 5
user.points += 3
if points.date < date.today():
points.chances_used = 1
points.date = date.now()
user.points += 3
Though thinking about it now this means you would lose record of where a users points came from because I am using user.points to store the actual value.
You might wanna just add a row to the table every-time you check instead. And get the data instead of something like db.get(user_points).filter(user.id=1, points_type=3), just get the users latest for that point_type. And use that row to check the date and get the chances_used but save your data into a new row.
You could create a dedicated struct or function to handle scores. From what I see in the code in order to apply points to the user you would need to pass a few information such as userDB, UserId, points_type and points, the same what you already have in the code. If you want the function to be stateless and available everywhere you can define it in dedicated userpoints package and simply export:
func SendPointsTo(db userDB, userId, pointsType, pointsCount int) error {
err := d.insertScoreTx(userId, pointsType, pointsCount )
if err != nil {
return errors.New("Sending points failed")
}
return nil
}
Don't mix logic of creating user with some other actions - it introduces confusion and breaks SRP. It's better to make a dedicated createUserHandler which performs chains of steps defined by the contract. A part of the contract would then be additional SendPointsTo func call.
Is it what you mean? If not can you be more preciseful in problem definition?
For this question which of the session or cookie will do the desired job ?
I'm intrigued by the results after triggering a $_Session because I thought it only triggered when a visitor accessed the website.
In one day I had 378 accesses to my website or to the server that hosts my website. The whole world is trying to access my website while it is not online (12 countries). But that is not the problem at all!
In less than 30 min there were 20 new visits from 6 different countries. Is it normal ?
Why are these accesses to my website counted when the statistics collected by my host only give me the reloads of pages that I perform (about 14)?
Does the Host already filter bot ?
But that doesn't explain why I get about 200 results from my own ip address when I deleted all windows to my website between tonight and this morning .
Also the increment is multiplied by two instead of being incremented by 1 .
How can I count the number of anonymous visitors every 24 hours who access my website?
Can I limit the duration of a session to 10 seconds ? Does this restriction can filter bots ?
I try to store in one row of specific table :
the auto-increment Id
the unix time of day
the number of visit incremented each time.
for now that's all I need.
Currently the main problem is the increment $_SESSION thats increment by 2 instead of 1 :
function StartSession(){
global $wpdb;
session_set_cookie_params(0, '/', '.website.com');
session_start();
if ( !isset( $_SESSION['visitor'] ) ) {
$countuser = $_SESSION['visitor'] = 1;
}else{
$countuser = ++$_SESSION['visitor'];
}
$users_analytics = $wpdb->prefix . 'antics_users';
$timeanonymesession = time();
$wpdb->insert(
$users_analytics,
array(
'datesession' => $timeanonymesession,
'totalusers' => $countuser
//'iploc' => $new_ip
)
);}
add_action('init', 'StartSession', 1);
If I use a cookie instead of a session, I think I'll have the same problem with incrementing, right?
Edit : in one day the *_SESSION have registered in database more than 1500 visit.
I don't understand why I have so many connection with $_SESSION whereas My Host registered only between 10 and 15 sessions by day (that normally corresponding to my reload page during developpement) ?
I am using caffeine cache and looking for a way to update the value in cache without changing its expire time.
The scenario is that I am using cache for speed up data loading. A 5 seconds' delay of data changing is acceptable while I expect returns to be fast. Besides, I want these cache to expiry after 1 day of its first hit to avoid unnecessary memory use.
Thus, I want every cached keys lasts for one day but its value is updated every 5 second.
The refreshAfterWrite method seems to be close but its first returned value after refresh duration is still the old one. This is not ideal for me because the duration between two hits can be hours. In this case I still want a relatively new result (no more than 5 seconds).
So I am trying to manually updating each key.
Firstly I built a cache with 24 hours' expire time in this way:
cache = Caffeine.newBuilder()
.expireAfterWrite(24, TimeUnit.HOURS)
.build();
Then I wrote a scheduled task per 5 seconds which iterate keys in cache and do following:
cache.asMap().computeIfPresent(key, mapperFunction);
Then I checked the age of the key:
cache.policy().expireAfterWrite().get().ageOf(key)
However, the age is not growing as expected. I think the computeIfPresent method is considered as a "write" so that the expiry time is also updated.
Is there a way to do the value update without change its expire time in caffeine?
Or any other approach for my scenario?
A write is the creation or update of a mapping, so expireAfterWrite is not a good fit for you. Instead you can set a custom expiration policy that sets the initial duration and does nothing on a read or update. This is done using expireAfter(Expiry), such as
LoadingCache<Key, Graph> graphs = Caffeine.newBuilder()
.expireAfter(new Expiry<Key, Graph>() {
public long expireAfterCreate(Key key, Graph graph, long currentTime) {
return TimeUnit.HOURS.toNanos(24);
}
public long expireAfterUpdate(Key key, Graph graph,
long currentTime, long currentDuration) {
return currentDuration;
}
public long expireAfterRead(Key key, Graph graph,
long currentTime, long currentDuration) {
return currentDuration;
}
})
.build(key -> createExpensiveGraph(key));
Need some opinion/help around one use case of KStream/KTable usage.
Scenario:
I have 2 topics with common key--requestId.
input_time(requestId,StartTime)
completion_time(requestId,EndTime)
The data in input_time is populated at time t1 and the data in completion_time is populated at t+n.(n being the time taken for a process to complete).
Objective
To compare the time taken for a request by joining data from the topics and raised alert in case of breach of a threshold time.
It may happen that the process may fail and the data may not arrive on the completion_time topic at all for the request.
In that case we intend to use a check that if the currentTime is well past a specific(lets say 5s) threshold since the start time.
input_time(req1,100) completion_time(req1,104) --> no alert to be raised as 104-100 < 5(configured value)
input_time(req2,100) completion_time(req2,108) --> alert to be raised with req2,108 as 108-100 >5
input_time(req3,100) completion_time no record--> if current Time is beyond 105 raise an alert with req3,currentSysTime as currentSysTime - 100 > 5
Options Tried.
1) Tried both KTable-KTable and KStream-Kstream outer joins but the third case always fails.
final KTable<String,Long> startTimeTable = builder.table("input_time",Consumed.with(Serdes.String(),Serdes.Long()));
final KTable<String,Long> completionTimeTable = builder.table("completion_time",Consumed.with(Serdes.String(),Serdes.Long()));
KTable<String,Long> thresholdBreached =startTimeTable .outerJoin(completionTimeTable,
new MyValueJoiner());
thresholdBreached.toStream().filter((k,v)->v!=null)
.to("finalTopic",Produced.with(Serdes.String(),Serdes.Long()));
Joiner
public Long apply(Long startTime,Long endTime){
// if input record itself is not available then we cant use any alerting.
if (null==startTime){
log.info("AlertValueJoiner check: the start time itself is null so returning null");
return null;
}
// current processing time is the time used.
long currentTime= System.currentTimeMillis();
log.info("Checking startTime {} end time {} sysTime {}",startTime,endTime,currentTime);
if(null==endTime && currentTime-startTime>5000){
log.info("Alert:No corresponding record from file completion yet currentTime {} startTime {}"
,currentTime,startTime);
return currentTime-startTime;
}else if(null !=endTime && endTime-startTime>5000){
log.info("Alert: threshold breach for file completion startTime {} endTime {}"
,startTime,endTime);
return endTime-startTime;
}
return null;
}
2) Tried the custom logic approach recommended as per the thread
How to manage Kafka KStream to Kstream windowed join?
-- This approach stopped working for scenarios 2 and 3.
Is there any case of handling all three scenarios using DSL or Processors?
Not sure of we can use some kind of punctuator to listen to when the window changes and check for the stream records in current window and if there is no matching records found,produce a result with systime.?
Due to the nature of the logic involve it surely had to be done with combination of DSL and processor API.
Used a custom transformer and state store to compare with configured
values.(case 1 &2)
Added a punctuator based on wall clock for
handling the 3rd case
I am reading stripe documentation and I want to fetch the first charge of the a customer. Currently I am doing
charge_list = Stripe::Charge.list(
{
customer: "cus_xxx"
},
"sk_test_xxxxxx"
)
first_charge = charge_list.data.last
Since stripe api returns the charges list in sorted order with the most recent charges appearing first. But I don't think it is a good approach. Can anyone help me with how can I fetch the first charge by a customer or how can I sort the list with descending order of created date so that I could get the first object from the array.
It seems there is no reverse order sorting feature in stripe API.
Also remember the first charge may not be on the first page result set, so you have to iterate using #auto_paging_each.
A quick possible solution:
charge_list = Stripe::Charge.list(
{customer: "cus_xxx", limit: 100 }, # limit 100 to reduce request
"sk_test_xxxxxx")
first_charge = nil
charge_list.auto_paging_each {|c| first_charge = c }
You may want to persist the result somewhere since it is a heavy operation.
But the cleanest solution IMO would be to store all charge records into your DB and make subsequent queries against it.