From Tiers to Percentiles: Introducing a Big New Change to Criticker

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
quartier
Posts: 31
1519 Ratings
Your TCI: na
Joined: Sun Jun 06, 2010 7:26 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by quartier »

love it! :D
Seriously... this site just get better.

But my regenerate PSIs button are not working.

Image

jeff_h
Posts: 12
1422 Ratings
Your TCI: na
Joined: Wed May 02, 2007 12:54 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by jeff_h »

After the switch, all of my PSIs for games are at 9.9999

mpowell
Posts: 3851
1201 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

quartier wrote:love it! :D
Seriously... this site just get better.

But my regenerate PSIs button are not working.


Okay, we'll check that out! Thanks for reporting the problem, and glad you like our progress.

mpowell
Posts: 3851
1201 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

jeff_h wrote:After the switch, all of my PSIs for games are at 9.9999


Okay, that definitely doesn't sound right. We'll check it out and see what we can find.

mpowell
Posts: 3851
1201 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

movieboy wrote:If we do switch, will it be possible to switch back if we don't like it?


Yes, you'll be able to switch back, if you really want to. But, your TCIs and PSIs will have to be reset again, and this time they won't be fully regenerated. Because of the strain on the server, we don't want to get into a situation where users are switching back-and-forth. Of course, if you're a sponsor of the site, switching back and forth is less of a problem, since full, daily generations of PSIs and TCIs are one of the main benefits of sponsorship.

Ramy
Posts: 86
1485 Ratings
Your TCI: na
Joined: Sat Aug 08, 2015 5:46 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by Ramy »

Found a little bug. When I regenerate my PSIs in the full database, the PSI for every movie becomes 30.

90sCoffee
Posts: 173
2242 Ratings
Your TCI: na
Joined: Sun Feb 17, 2013 5:15 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by 90sCoffee »

Same, that's actually a big bug because that's what I mainly use Criticker for lol, the PSI numbers. They've all gone to either 30, 35, or 40 as soon as I click regenerate PSI on a film in my wishlist.

mattorama12
Posts: 887
3086 Ratings
Your TCI: na
Joined: Mon Feb 13, 2012 3:05 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mattorama12 »

This is a great update. I'm not sure it's that much more intuitive (but I guess I never found tiers difficult to grasp), but I love it for the improved granularity. I would expect that now, the new source of confusion for new users will be something similar to what we had before but will sound like this: "why is this movie that I rated 95 only in my 65th percentile?"

Anyway, one thing that I think is lost with the new system is a quick visual way to see things in the "all rankings" view. When we had tiers, there was a divider for each one. Having a divider for every percentile would be too busy, but I think it would be nice to have a divider for every 10th percentile or even 5th percentile.

Zarkon
Posts: 23
1056 Ratings
Your TCI: na
Joined: Sat Jun 03, 2006 8:10 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by Zarkon »

mpowell wrote:You can see the Percentile by opening the info box for the film. And if you hover over the percentile, you can see how it was calculated. Incidentally, you can do this anywhere on the site, where a percentile appears.

The info box isn't accessible on Profile pages, and the mouseover popup doesn't work on the scores there, either.

Honestly, the mouseover popup is more busy, slower to get information out of (especially with the fade-in), and less mobile-friendly compared to the always-visible "T#" numbers that it replaced.

Other things:
- "A score of ... puts this in the ... percentile" line is missing on films with scores at the very bottom (sub-1.0 percentile, maybe).

- Seconding the request to add visual demarcations for percentiles in the All Rankings page.

- I think it's about time the score distribution chart is changed to a bar graph or histogram. The line graph isn't appropriate when most users only use a subset of the 0-100 scoring.

- On that same page, it would be nice to have a percentile (y-axis) vs score (x-axis) graph.

trippingly
Posts: 33
1045 Ratings
Your TCI: na
Joined: Wed Jul 17, 2013 5:18 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by trippingly »

I think it's a really great idea to start using percentile rank. More granularity should make for more accurate predictions (and I already found them very accurate), and I think people may find it easier to understand.

I suggest making one important change to the formula used to calculate percentage rank. Right now, to calculate the percentile rank of a given score, Criticker is using the formula:

# of films with a lower score / total # of films ranked

(multiplied by 100 and rounded down)

And here's the formula for percentile rank from WIkipedia:

Image

where cℓ is the count of all scores less than the score of interest, ƒi is the frequency of the score of interest, and N is the number of examinees in the sample.


In Criticker terms, that would be:

(# of films with a lower score + half the # of films with the score in question) / total # of films ranked

(multiplied by 100 and rounded down)

Here's why that part in bold is important. Say you have two users who have each rated 10 films...

A: 0 20 30 40 50 50 70 80 90 100

B: 0 0 0 0 50 50 50 50 80 100

What percentile rank should a score of 50 work out to?

Both users have four films rated lower than 50, so without the part in bold, a score of 50 works out to the 40th percentile for both of them (4 films ranked lower / 10 films total).

But notice that for user A, the range of films with a score of 50 falls right in the middle of the overall range. Reflecting that, if you use the full formula, a score of 50 for user A works out to the 50th percentile ((4 + 1) / 10).

Whereas the range of films that user B has given a 50 extends higher. Reflecting that, a score of 50 for user B works out to the 60th percentile ((4 + 2) /10).

Or what about a score of 0?

Without the part in bold, every user's lowest score will work out to the 0th percentile, no matter how many films the user has given that score.

But user A has only one film with a score of 0 at the very bottom of their rankings, whereas user B has given a 0 to a wider range of films. Maybe user B is a really tough critic, or maybe they just decided not to split hairs when it comes to films they didn't like. Either way, whereas a 0 from user A is rare and therefore pretty damning, a 0 from user B is less so. Reflecting that, if you use the full formula, a score of 0 for user A works out to the 5th percentile ((0 + 1/2) / 10), and a score of 0 for user B works out to the 20th percentile ((0 + 2) / 10).

Basically, the full formula gives you the midpoint (as opposed to the lower bound) of a given range of films... the point where the given range is centered within the overall range of all the films the user has rated.

Or here's another way to look at it... We said that for user B, a score of 50 works out to the 60th percentile. That's like saying that if user B was no longer allowed to give the same score to multiple films and had to figure out a way to rank them in order of preference without any ties, the films that used to have a score of 50 would have an average rank of 60%, would be centered around the 60% mark in the lineup.

Hope that's helpful. Definitely not an expert on this stuff, but I believe that without that part of the formula you lose valuable information and that the full formula will make for more meaningful comparisons from one user to another.

Post Reply