Return To: Board Index | Feature Requests

Criticker ranking system has one enormous flaw

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
MTR
Posts: 1
0 Ratings
Your TCI: na
Joined: Fri Jul 12, 2019 7:13 am

Criticker ranking system has one enormous flaw

Postby MTR » Thu Aug 15, 2019 5:29 pm

Imagine we have hypothetical user A who has only ever ranked 26 movies, each one with different rating from 75 to 100.

Then we have user B who has ranked those 26 movies and gave them exactly the same ratings as user A. But on top of that, user B also ranked additional 75 movies, each one with different rating from 0 to 74.

TCI of user A and user B is over 38 :!:

But quite obviously, it should be 0 :!:


So... I don't know how viable is this outside aforementioned example, but to get the TCI at 0, you would need to calculate percentiles only among shared rated titles. It seems it would also add a lot of new calculations to be made, so it might not be viable for not so big site, but leaving it as it is certainly a bad idea.

djross
Posts: 1134
4193 Ratings
Your TCI: na
Joined: Sun Apr 16, 2006 12:56 am

Re: Criticker ranking system has one enormous flaw

Postby djross » Tue Aug 20, 2019 2:00 am

MTR wrote:But quite obviously, it should be 0 :!:


This is not obvious at all since it assumes that what matters is the numerical score rather than the overall distribution: if a user's lowest score is 75, then that's the worst movie they've ranked. If instead of 26 movies they ranked 2600 movies, the situation would still be the same, so if it's just that they haven't ranked any of the bad movies they've seen yet, then the best solution is to rank more movies. The system's fine.

PrestoBix
Posts: 35
951 Ratings
Your TCI: na
Joined: Thu Aug 20, 2015 6:48 am

Re: Criticker ranking system has one enormous flaw

Postby PrestoBix » Mon Sep 09, 2019 1:55 pm

Yeah this totally isn't an issue at all.

livelove
Posts: 497
66 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Postby livelove » Sun Sep 15, 2019 11:58 pm

@MTR: welcome to the forum.

The users who replied to you didn't get your point, which is a point well taken.
Be assured, you have raised a legitimate issue that is a regularly resurfacing topic of discussion here, see:
viewtopic.php?f=10&t=7255&p=66438#p66438
viewtopic.php?f=8&t=7143&p=68661#p66563

amazedemon
Posts: 17
2321 Ratings
Your TCI: na
Joined: Mon Aug 18, 2014 11:29 am

Re: Criticker ranking system has one enormous flaw

Postby amazedemon » Tue Sep 17, 2019 9:38 pm

MTR wrote:TCI of user A and user B is over 38 :!:

Surely your assumption would mean the TCI is ~28, not 38 (or 37) as the 26 films that are rated identically contribute 26 zeros to the average.

That however is nit-picking. My main point, or rather question, is whether the TCI's (in general) are mutually commutative. That is to say would User A's TCI for User B be 0, and User B's TCI for User A be 28.

My TCIs for people who have commented are:
@MTR: not enough ratings
@djross: 3.5270
@PrestoBix: 2.9076
@PedroPT: 2.1738
@livelove: not enough ratings

Is your TCI with me the same?

NB. I'm still on the old tier system.

livelove
Posts: 497
66 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Postby livelove » Tue Sep 17, 2019 10:42 pm

amazedemon wrote: My […] question, is whether the TCI's (in general) are mutually commutative.
My understanding is that this is the case.

amazedemon wrote:would User A's TCI for User B be 0, and User B's TCI for User A be 28.
following the above, no the TCI would be the same for both users (in both directions)

amazedemon wrote: @livelove: not enough ratings
Is your TCI with me the same?
your TCI with me currently is: 21.1429
(you would have to lower the "Films in Common Minimum" value to get a TCI for me)

livelove
Posts: 497
66 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Postby livelove » Tue Sep 17, 2019 10:51 pm

MTR wrote: you would need to calculate percentiles only among shared rated titles
cross-reference:
the same proposal has also been made in this thread:

Take into account the fact that not all users have seen the same films only

uvlalid wrote:I propose that when calculating the TCI of a user, only the films that both users have seen are used in the equation. So instead of calculating the percentile scores of films based on all the films seen by a user, the percentiles should be based only on all the films both users have in common.

To give an example scenario, imagine there are two users:

User 1
Films watched - Score
Film A - 50
Film B - 0
Film C - 100

User 2
Films watched - Score
Film A - 100
Film B - 50
Film D - 0

Using my suggested system to calculate the TCI of User 2 for User 1, only Films A and B will be used which for both users will have percentile scores of 100 and 0 respectively in that case.

amazedemon
Posts: 17
2321 Ratings
Your TCI: na
Joined: Mon Aug 18, 2014 11:29 am

Re: Criticker ranking system has one enormous flaw

Postby amazedemon » Wed Sep 18, 2019 8:56 pm

Honestly I'm okay with the way my PSI appear at the moment as they are usually within a 2 point margin. Though there are many older films/ Japanese films with quite high PSI's that I have very little interest or motivation in watching. (Yes I know I'm a philistine).

Sometimes I just like playing around with data, and asking what if?
I'm currently playing around with an extended version of @uvlalid's example to see where that leads.

livelove
Posts: 497
66 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

Re: Criticker ranking system has one enormous flaw

Postby livelove » Wed Sep 18, 2019 9:04 pm

amazedemon wrote:I'm currently playing around with an extended version of @uvlalid's example to see where that leads.
great, please share if you come across something interesting


Return to “Feature Requests”