When Ratings Vary Widely, Averages Can't Be Trusted

[oplars]

I'm leaning towards the idea that movie ratings should be given as a range, like 65-80 or something. (EDIT: see my next post for a moderation of this viewpoint)
If I observe that the ratings from the top 10 TCIs for a particular movie are highly inconsistent—ranging from love to hate to indifference—I conclude that my own reaction to the movie could be equally unpredictable. I.e. that there are an equal chance that I will love or hate or be indifferent to the movie. So I've started looking at how the ratings are spread instead of the PSI. Let's say I am interested in a certain movie with a PSI below 50th percentile. I then notice that there are two dark green ratings among the top TCIs. These two guys were smart not to listen to their top TCIs opinions on the movie since they were obviously way off. In cases where the range of ratings of the top TCI is wide, this in itself indicates that the average - in these specific cases - cannot be relied upon as a reliable predictor of any of these individual's opinion.
Therefore, the spread should be taken into consideration when offering a PSI.
Below is one of the extreme examples of what I mean:

[90sCoffee]

I'd disagree with this, I like having a tangential number I can sort. Also would probably be a headache in terms of database computations.

I think for now you can assume your actual PSI for a film is a rough suggestion and +/- whatever depending on your rating scale.

[oplars]

You are right, showing ranges isn't the way to go, I think neither. But it would be a significant improvement if it was showed that the PSI is very uncertain is these cases.
A way cooler idea would be to use a bright color to show when the PSI is super reliable. This would intuitively make sense to new users. A very pale green would suggest that there is a chance that you will like this movie. Medium green: a fair chance. Strong green: Almost certain.
The colors brightness could also be influenced by the quantity of good TCIs.
(High ratings from a few good TCIs + the rest of the Top 10 ratings are all over the place > pale green. ALL Top 10 are good TCIs + they ALL give a good rating > strong green)

Movie recommendation engines all lack reliability indicators, which is their weakest point.
(Remember how in old days the newspaper would just report if it was going to be rain? Not how sure they were about that)
Criticker provides visible data to make better judgments. However, it shouldn't be difficult for the system to calculate the value of a PSI since it's something a human can determine at a glance. It almost already does so in stats like the one in the former post.

[Bryce74]

I think that is a VERY good idea.
It would also give people more confidence in Criticker.
I heavily rely on Criticker, combined with IMDB and often Criticker gives great recommendations (better than Chatgpt, I tried) but still too often Criticker disappoints me. By indicating how reliable each PSI score is, Criticker would guide my decisions much better.

[Bryce74]

I think that is a VERY good idea.
It would also give people more confidence in Criticker.
I heavily rely on Criticker, combined with IMDB and often Criticker gives great recommendations (better than Chatgpt, I tried) but still too often Criticker disappoints me. By indicating how reliable each PSI score is, Criticker would guide my decisions much better.

[geohawk]

One thing that earlier recommender systems -- most long since wiped out by the asteroid that killed most interesting web2 sites -- frequently had was a confidence indicator, usually driven by some combination of rating variability and the "similarity" of recommending users to the user. In other words, the more uniform the ratings of my most similar users, the more confident the recommender would be in its prediction.

At Criticker, it only takes three users amongst a pool of your 1000-2000 most similar viewers to get a prediction. Requiring more reduces the ability of the site to make recommendations for lesser seen movies (or television shows and television). On the Games side of the site, it can be worse, where there are too few users to exclude people from the recommending pool based on differences of opinion and even your worst TCIs can be used to calculate your prediction.

A possible confidence indicator for a prediction would probably need to look at rating variability, how many ratings were made to make the prediction, and where in the ranking of your TCIs the ratings came from. Ten 90 percentile scores from your top 100 TCIs would presumably lead to greater confidence than a prediction derived from TCIs number 1998, 1999, and 2000.

[joseywales]

You should get a girlfriend bud

I ignore my PSI for the most part. I rate movies based on things like year they came out and similar movies around the same time (maybe a 5 year span or so), genre in general and in same time period, other wack comparisons I might make (the directors other films etc).

Some of my best kumpels have totally different tastes in film than I do, but I still respect their opinion so I look for their reviews first.

Funny part is... my PSI is usually bang on within 2-3 points or off by like 20.