This question is more related to logic than any programming language. If the question is not apt for the forum please do let me know and I will delete this.
I have to write a logic to calculate scores for blogs for a Blog Award website. A blog may be nominated for multiple award categories and is peer-reviewed or rated by a Jury on a -1 to 5 scale (-1 to indicate a blog they utterly dislike). Now, a blog can be rated by one or more Jurors. One criterion while calculating final score for a blog is that if a blog is rated positively by more people it should get more weightage (and vice-versa). Similarly a blog rated -1 even by one Juror should have its score affected (-1 is sort of a Veto here). Lastly, I also want to have an additional score based on the Technorati rank of the blog (so that the final score is based on a mix of Juror rating + Technorati ranking).
Example: A blog is rated in category A by total 6 Jurors. 2 rate it at 3, 3 rate it at 2 and 1 rate it at 4. (I used to calculate the score as (2*3 + 3*2 + 1*4)/6 = 16/6 = 2.67 to get weighted average but I am not satisfied with this, primarily because it doesn't work well when a Juror rating is -1. Moreover, I need to add the Technorati ranking ranking criteria too) .
Could you help me decide the best way to calculate the final scores (keeping the rating method same as above as that cannot be changed now)?
If you want to weight the effect of a -1 rating more strongly, use the same average score calculation but substitute -10 whenever you see -1. You can choose a value other than -10 if you don't want a negative rating to weight as strongly.
You might look at using the lower bound of the Wilson score interval for your ratings.
See http://www.evanmiller.org/how-not-to-sort-by-average-rating.html for more details. Although, there, it is used for the simpler Bernoulli case.
The gist is if you have a lot of ratings you have a higher degree of confidence in your scoring. You can then combine the scores from your local ratings and the Technorati ratings, by weighting the scores by the number of voters locally and on Technorati.
As for wanting a single -1 vote to have high impact, just remap it to a large negative value proportional to your desired impact before feeding it into your scoring formula.
Calculating a score based on votes will be pretty easy. Adding the technorati rank will be the tricky part.
I made a quick script that calculates some scores based on this algorithm
score = ( vote_sum - ( vetos * veto_weight ) ) / number_of_votes
you can change the url paramters to get different values
There are a lot of ties, so maybe you could use technorati blog rank as a tie breaker
you could internally work with scores from 0 to 6. Just do a shift by one, calculate the score and shift back. I guess the -1 has some disrupting effekt on your calculation.
Related
I'm working on a rating algorithm. I have a set of exercises. They are all categorized in levels (1 = easiest, 5 = hardest).
Users get shown two exercises and should decide which one is harder or if both are equal. Based on user ratings, the levels should get adjusted.
What I've done:
I experimented with the Elo rating.
My Questions:
Are there any better algorithms for doing this use case? (found nothing so far)
Thanks in advance and cheers.
Toby
I would try to solve the problem in a simple yet (I hope) effective way.
First, you only update an exercise rating when the vote is different that what the system actually expects. From now on, I will only considers the cases where the user output differs from what the system actually expects.
Second, I would give more weight to the votes where the two levels have a big difference. A wrong expectation on two esercises with rating 2 and 3 should have less impact than a wrong expectation on two exercises with rating 1 and 5.
That said, my algorithm would be along the lines of:
1- A constant percentage is set, let's call it increment. It establishes the percentage of impact that a vote has, and can be modified along the way based on the number of users.
2- For an "unexpected" vote, I would calculate the difference between the original levels (minimum of 1).
diff = max(1, abs(ex1.level - ex2.level))
3- I would update each exercise rating by a percentage, based on the multiplication of increment and diff.
if (ex1 level expected bigger)
ex1.rating = ex1.rating + diff*increment;
else
ex1.rating = ex1.rating - diff*increment;
Rating would be a float, and level would be the rounding of rating:
ex1.level = round(ex1.rating)
Example:
let's set increment = 0.1. exA, with a rating of 2.0 and level 2 is compared with exB, rating of 3.0 and level 3.
The first user selects exB as the hardest. Nothing changes, because it is the result expected by the system.
The second user selects exA. It is not the expected result. The difference between the two exercises is 1, so the rating is modified by a factor 1*0.1 = 0.1,
resulting in a exA.rating = 2.1 for exB.rating = 2.9
I've to come up with an algorithm which would determine jumps and changes of a person's personal scores. Just imagine that every day a person participates in a competition and the scores are recorded on daily basis. Now my task is to come up with a person's performance ratio based on provided time interval. For example: A person scored 7 yesterday, scored 6 today, which means the performance is negative: -1 .
My current solution:
I've two collection of numbers which represents scores, where each element is a daily score of a person (three days):
dataFor2014-07-11/13 = {6,6,6}
dataFor2014-07-13/15 = {6,3,5}
double personsScores = AVG(dataFor2014-07-13/15) - AVG(dataFor2014-07-11/13);
Output: 4.6 - 6 = -1.3 (person's performance for past six days is negative.)
Do you think this is reasonable algorithm? Do you have any suggestions how can I improve it and recommend any better solution?
I think this is more a mathematical problem and the Math forum would probably better to ask.
Probably, convolution filters are the way to go. This is a technique to make a 'graph' through some points (the individual scores).
I'm trying to develop a rating system for an application I'm working on. Basically app allows you to rate an object from 1 to 5(represented by stars). But I of course know that keeping a rating count and adding the rating the number itself is not feasible.
So the first thing that came up in my mind was dividing the received rating by the total ratings given. Like if the object has received the rating 2 from a user and if the number of times that object has been rated is 100 maybe adding the 2/100. However I believe this method is not good enough since 1)A naive approach 2) In order for me to get the number of times that object has been rated I have to do a look up on db which might end up having time complexity O(n)
So I was wondering what alternative and possibly better ways to approach this problem?
You can keep in DB 2 additional values - number of times it was rated and total sum of all ratings. This way to update object's rating you need only to:
Add new rating to total sum.
Divide total sum by total times it was rated.
There are many approaches to this but before that check
If all feedback givers treated at equal or some have more weight than others (like panel review, etc)
If the objective is to provide only an average or any score band or such. Consider scenario like this website - showing total reputation score
And yes - if average is to be omputed, you need to have total and count of feedback and then have to compute it - that's plain maths. But if you need any other method, be prepared for more compute cycles. balance between database hits and compute cycle but that's next stage of design. First get your requirement and approach to solution in place.
I think you should keep separate counters for 1 stars, 2 stars, ... to calcuate the rating, you'd have to compute rating = (1*numOneStars+2*numTwoStars+3*numThreeStars+4*numFourStars+5*numFiveStars)/numOneStars+numTwoStars+numThreeStars+numFourStars+numFiveStars)
This way you can, like amazon also show how many ppl voted 1 stars and how many voted 5 stars...
Have you considered a vote up/down mechanism over numbers of stars? It doesn't directly solve your problem but it's worth noting that other sites such as YouTube, Facebook, StackOverflow etc all use +/- voting as it is often much more effective than star based ratings.
I am writing an e-commerce engine that has a reputation component. I'd like users to be able to review and rate the items, and be able to rate the review.
What is the best algorithm to use to sort items based on "best" reviews? It has to be ranked by the amount of quality reviews it gets by the people who give the best reviews. I'm not sure how to translate this to algorithm.
For instance, I need to be able to compare between an item that has 5 stars from many people with low reputation, against another item that has 3 stars from a few people with high reputation.
To add to the complexity, some users may have written many reviews that are rate high / low by others, and other users may have written few reviews, but rated very high by other users. Which user is more reputable in this case?
If you know the reputations of the users, then you might use a UserScore for each user such as the one that Stackexchange uses.
UserScore = Reputation >= 200 ? 0.233 * ln(Reputation-101) - 0.75 : 0 + 1.5
Then you find the value of an item by summing up the user scores with the stars as weights:
ItemScore = \sum_i UserScore_i * Weight[Star_i]
where i is the index for the votes and Weight is the array involving the weights of stars. For example, it can be [-2 -1 0 1 2] for a voting system of 5 stars. And one note is that you may change the weight of the 3 stars to be +eps if you want the items with only 3 stars to come before the items which are not evaluated.
You may change 200 and all other constants/weights accordingly to your needs.
I will try to answer your question:
I think the trick is to weight out the people with different reputation, for instance:
A person with a reputation 2 has a vote that is 3x as heavy as the vote of another person with a lower reputation of 1. That relationship between people of different reputation is really up to you and how much you want to have the overall rating dependent on the ratings of people with low reputation. The higher the vote weight of a person with high reputation compared to a vote of a low reputation person, the less the overall reputation will change due to the low reputation votes.
So each person will have a weight let's say w_i, w_j etc.... and then the over all rating will be the weighted average of all:
example of overall rating of votes from two different persons i and j = (w_i*r_i)+(w_j*r_j)/(w_i + w_j)
where r_i, r_j are the ratings of person i and person j respectively.
To get the value of the weights of each person, you can for instance take the number of stars that person.
A good resource would be the following page:
http://en.wikipedia.org/wiki/Weighted_mean
I'd like to rank my stories based on "controversy" quotient. For example, reddit.com currently has "controversial" section: http://www.reddit.com/controversial/
When a story has a lot of up and a lot of down votes, it's controversial even though the total score is 0 (for example). How should I calculate this quotient score so that when there's a lot of people voting up and down, I can capture this somehow.
Thanks!!!
Nick
I would recommend using the standard deviation of the votes.
A controversial vote that's 100% polarised would have equal numbers of -1 and +1 votes, so the mean would be 0 and the stddev would be around 1.0
Conversely a completely consistent set of votes (with no votes in the opposite direction) would have a mean of 1 or -1 and a stddev of 0.0.
Votes that aren't either completely consistent or completely polarised will produce a standard deviation figure between 0 and ~1.0 where that value will indicate the degree of controversy in the vote.
The easiest method is to count the number of upvote/downvote pairings for a given comment within the timeframe (e.g. 1 week, 48 hours etc), and have comments with the most parings appear first. Anything more complex requires trial-and-error or experimentation on the best algorithm - as always, it varies on the content of the site and how you want it weighted.
Overall, it's not much different than a hotness algorithm, which works by detecting the most upvotes or views within a timeframe.
What about simply getting the smaller of the two values (up or down) of a point in time? If it goes up a lot and goes down a little, or the other way around it, is not controversial.
If for example the items has 10 ups and 5 downs, the "controversiality level" is 5, since there is 5 people disagreeing about liking it or not. On the other hand if it has either 10 ups or 10 downs, the "controversiality level" is 0, since no one is disagreeing.
So in the end the smaller of both items in this case defines the "hotness" or the "controversiality". Does this make sense?
// figure out if up or down is winning - doesn't matter which
if (up_votes > down_votes)
{
win_votes = up_votes;
lose_votes = down_votes;
}
else
{
win_votes = down_votes;
lose_votes = up_votes;
}
// losewin_ratio is always <= 1, near 0 if win_votes >> lose_votes
losewin_ratio = lose_votes / win_votes;
total_votes = up_votes + down_votes;
controversy_score = total_votes * losewin_ratio; // large means controversial
This formula will produce high scores for stories that have a lot of votes and a near 50/50 voting split, and low scores for stories that have either few votes or many votes for one choice.