Criticker ranking system has one enormous flaw

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
livelove
Posts: 759
67 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Post by livelove »

amazedemon wrote:I'm currently playing around with an extended version of @uvlalid's example to see where that leads.
great, please share if you come across something interesting

MTR
Posts: 21
34 Ratings
Your TCI: na
Joined: Fri Jul 12, 2019 7:13 am

Re: Criticker ranking system has one enormous flaw

Post by MTR »

[spoiler]
djross wrote:
MTR wrote:But quite obviously, it should be 0 :!:


This is not obvious at all since it assumes that what matters is the numerical score rather than the overall distribution: if a user's lowest score is 75, then that's the worst movie they've ranked. If instead of 26 movies they ranked 2600 movies, the situation would still be the same, so if it's just that they haven't ranked any of the bad movies they've seen yet, then the best solution is to rank more movies. The system's fine.


You haven't understood my post one a bit. :)

PrestoBix wrote:Yeah this totally isn't an issue at all.


Thank you for your "input". :)

amazedemon wrote:
MTR wrote:TCI of user A and user B is over 38 :!:

Surely your assumption would mean the TCI is ~28, not 38 (or 37) as the 26 films that are rated identically contribute 26 zeros to the average.


You either didn't understood my post one a bit or you simply don't understand TCI one a bit. :)

-------------------------------------------------------------------------------------------------------------------------------------------------------[/spoiler]

Anyway something that might interested livelove I guess. I made two simple examples and will post them in image form.

Example 1 (similar to my opening post) https://i.imgur.com/DB89dPA.png

Example 2 (showing that rating bad titles makes your TCI much worse than it should be with someone who only rates good titles) https://i.imgur.com/MKXF4bn.png

Example 3 (the best one yet, user A and user B have very similar tastes, but user A watches a lot of good movies that user B doesn't and user B watches a lot of bad movies, that user A doesn't) https://i.imgur.com/LYhMKF9.png

uvlalid
Posts: 8
1338 Ratings
Your TCI: na
Joined: Tue May 15, 2018 6:21 pm

Re: Criticker ranking system has one enormous flaw

Post by uvlalid »

MTR wrote:Example 1 (similar to my opening post) https://i.imgur.com/DB89dPA.png

Example 2 (showing that rating bad titles makes your TCI much worse than it should be with someone who only rates good titles) https://i.imgur.com/MKXF4bn.png

Example 3 (the best one yet, user A and user B have very similar tastes, but user A watches a lot of good movies that user B doesn't and user B watches a lot of bad movies, that user A doesn't) https://i.imgur.com/LYhMKF9.png

Nice figures MTR, thanks for taking the time to generate them. If it's not too much to ask, would you mind also adding some descriptive analyses by interpreting the results? It's hard for me to understand the figures and draw conclusions and would really appreciate it if you could explain them, so I'm thinking others might do too.

MTR
Posts: 21
34 Ratings
Your TCI: na
Joined: Fri Jul 12, 2019 7:13 am

Re: Criticker ranking system has one enormous flaw

Post by MTR »

I already added short descriptions to those examples, but I guess some longer interpretations would be nice anyway:

Example 1 - User A rates all the movies they watch, even those they found not worthy of watching till the end, those movies just get low rating. On the other hand User B rates only movies they finished, which means bad movies don't get any rating from them. However movie taste of User A and User B is identical. (I played a bit with actual ratings, so only percentiles are identical and not ratings, to not get another nonsensical

"it assumes that what matters is the numerical score rather than the overall distribution"
:roll:

Current TCI between User A and User B would be ~0.33 which would suggest that on average their movie ratings differ a whopping 33%. Just from this number I would say their tastes are far from similar.
On the other hand new-TCI would be 0 (zero), which would suggest they always give movies the very same rating (in their respective scales). For all we know User A and User B might be the same person on different accounts.

----------------------------------------------------------------------------------------------------------------------------------------------------

Example 2 - This is just another take on Example 1 with some small changes, not really a good example when I look at it now as it is a bit redundant. Anyway, User A and User B seem to be swapped here. It's User A who doesn't rate bad movies (maybe they don't watch them till the end and thus don't want to rate them or just couldn't be bothered with rating them), whilst user B rates all movies, including the bad ones. Their tastes are rather similar, but not identical as in Example 1.

Current TCI would be ~0.54 indicating an enormous difference in ratings, 54% on average :!: :!: For what it's worth, judging from this number, those users should never go to the cinema together as their opinions on the movies almost never match.
Which we know is not true.
New-TCI would be ~0.13 showing fairly small 13% on average difference between ratings. From this number we can say User A and User B would be a great cinema-going buddies who could have a blast watching movies and discussing them after the screening.

----------------------------------------------------------------------------------------------------------------------------------------------------

Example 3 - User A and User B have really similar opinions about the movies they have both watched, however User A loves the cinematography and watches a lot of old classics which they rate very high. On the other hand User B treats movies as cheap entertainment, doesn't look for good movies on purpose, doesn't watch old classics, instead they watch a lot of blockbusters which User A would never watch on principle. User B rates those blockbusters honestly, some of the best ones get good ratings, but most of them get rather low ratings.

Current TCI would be ~0.48 which means their tastes are faaar from each other. User B will never get "old classics" in their suggestions to watch making them even more adamant in not watching "old classics", because they're "bad" movies.
New-TCI is ~0.05 which means their tastes almost match. User B will get "old classics" in their suggestions to watch, but they can still ignore them on their own volition on the basis of them being "old" and hard to get. Or they can go and watch them and discover a whole new world of cinematography.

----------------------------------------------------------------------------------------------------------------------------------------------------

Thread is few months old and I'm yet to see a counterexample, which makes me strongly believe that new way of calculating TCI would be superior to the current one. However new-TCI has one big disadvantage compared to current one - it would require a much more calculations. Instead of calculating percentiles for every user once, it would require calculating percentiles for every pair of users. Assuming there's 109240 members on Criticker (a number I took from the forums statistics), current TCI requires 109240 percentiles calculations, new-TCI requires 5966634180 percentiles calculations, which is over 50 thousand times more calculations. I don't know how much realistic that would be :|

Post Reply