From Tiers to Percentiles: Introducing a Big New Change to Criticker

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
mpowell
Posts: 3831
1200 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

If you've been using Criticker for any length of time, you know all about Tiers. Tiers are how we normalize rankings across the 0-100 ranking scale. Your score might be an 86, but if that score is in your 62nd percentile of rankings, that means it's a Tier 7. One person's 86 might be a rather low score, while another person might consider it quite high. Tiers account for that.

The concept isn't terribly difficult to grasp, but it's a little abstract. I don't know how many times we've had to address the phenomenon of "Missing Tiers". If you don't have a wide variety of rankings, certain "tiers" might not appear. Also, there have been a lot of users who would like to "assign" certain scores to certain tiers, which is just not possible, because... it's math. It's not "configurable".

Image

Of course, when you have to answer the same question hundreds of times, maybe it's your system that's the problem. At some point, we realized that we were always addressing these concerns in the same way: "Tiers are more understandable if you consider that they represent percentile groups". And then we thought... why aren't we just using percentiles?

So now, we are. In one of the biggest changes to Criticker since the website launched, we are shifting from Tiers to Percentiles.

From a conceptual standpoint, this isn't a huge leap. Tiers were representations of percentiles, anyway. But you'll notice a lot of changes at Criticker. The biggest will be in your TCIs. Because we're now averaging "Percentiles" instead of "Tiers", the TCIs will be much bigger numbers. The difference between Tiers maxed out at 9, but the difference between Percentiles can range up to 99. So, a good TCI will now be something like 18.435 as opposed to 1.843. We think this is cool, as it introduces more variation (rather: the variation is more visible).

Image

What happens to your recommendations? PSIs will improve slightly. We tested the new system against a wide variety of users, and in nearly every case, accuracy improved -- not a whole lot, but a little. (This testing is actually what gave birth to our recently-introduced Accuracy feature for sponsors.) It makes sense; percentiles are basically the same as tiers, but they're more refined. Which means that the resulting PSIs are, as well.

Image

If you love tiers, and you can't imagine Criticker's system without those "Tier 9" headers, don't worry. No one is going to force you to switch. Because they're so similar on the back-end, we're allowing tiers and percentiles to co-exist, at least for the foreseeable future. New users will see the percentile version, but nothing will change for existing users. However, you can switch the new system at any time, using a link your profile page. When you do, your TCIs and PSIs will have to be rebuilt. And as a one-time way to say "thanks" for being a member, we'll rebuild them fully for you, once you've initiated the switch.

That's all! We think this change is going to make Criticker much more accessible to the general public. People understand the idea of "percentiles" on a more intuitive level than "tiers"... which shouldn't be too surprising, since "percentiles" are a basic mathematical concept, and "tiers" are basically something we made up.

Let us know if you have questions, or if you discover any bugs (I'm sure there will be some). As always, we'll do our best to answer everything as quickly as we can.

zevel6
Posts: 42
4224 Ratings
Your TCI: na
Joined: Wed Mar 16, 2011 3:20 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by zevel6 »

Interesting changes.

I just changed from tiers to percentiles but where can you see the percentiles for movies you have ranked?
For example, under "all your rankings" in the past it would say the tier and then the films in that tier but now that has disappeared with no replacement.

mpowell
Posts: 3831
1200 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

You can see the Percentile by opening the info box for the film. And if you hover over the percentile, you can see how it was calculated. Incidentally, you can do this anywhere on the site, where a percentile appears.

Image

Kuroel
Posts: 2
2999 Ratings
Your TCI: na
Joined: Mon Oct 29, 2012 3:27 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by Kuroel »

I have quite a lot of titles ranked and I've never had any issues with the tiers. But is it still advisable to switch to the percentile system in the long run? At least, I was left under the impression that the tier ranking will eventually stop existing.

zevel6
Posts: 42
4224 Ratings
Your TCI: na
Joined: Wed Mar 16, 2011 3:20 pm

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by zevel6 »

mpowell wrote:You can see the Percentile by opening the info box for the film. And if you hover over the percentile, you can see how it was calculated. Incidentally, you can do this anywhere on the site, where a percentile appears.

Image


Thanks for that - after years of being on this site I never even knew this info box existed!
The precentile doesn't seem to be present for all titles though (see below).

Image

Also, there appears to be an issue with extracting the results since the change to percentiles (i get the following error message). Would it be possible to include the percentiles in the extract?

Image

Finally, is there a way to switch back to the tiers? I am regretting switching to percentiles as there seems to be a few other things missing (the above plus when ranking a movie it no longer says "not good" "good" etc based on the tier).

mpowell
Posts: 3831
1200 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

Kuroel wrote:I have quite a lot of titles ranked and I've never had any issues with the tiers. But is it still advisable to switch to the percentile system in the long run? At least, I was left under the impression that the tier ranking will eventually stop existing.


The eventual goal is to get everyone switched. It's going to be the version of Criticker which all new users get by default, and I can imagine that eventually we'll want to move completely to it. We're leaving tiers for now, because a lot of our long-term users associate their Criticker rankings pretty tightly with the existence of tiers, and we didn't want to shock anyone.

We really don't have any plans right now to get rid of the tiers... we're going to wait and see what the reaction is, and make up our minds a few months down the line. In general, though, I don't think there's any reason not to switch, as they're basically the same thing. Percentiles are just a little more refined.

mpowell
Posts: 3831
1200 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

zevel6 wrote:Thanks for that - after years of being on this site I never even knew this info box existed!

It's new, with the introduction of percentiles :D


zevel6 wrote:The precentile doesn't seem to be present for all titles though (see below).

Almost definitely a bug! We'll take a look very soon, and update here.

zevel6 wrote:Also, there appears to be an issue with extracting the results since the change to percentiles (i get the following error message). Would it be possible to include the percentiles in the extract?

Definitely a bug.

zevel6 wrote:Finally, is there a way to switch back to the tiers? I am regretting switching to percentiles as there seems to be a few other things missing (the above plus when ranking a movie it no longer says "not good" "good" etc based on the tier).

The little quips which appear after ranking a film should also be present... if they're not, it's likely another bug. (That's a hat trick of bugs for this post, zevel6!) If you want to switch back, until we've fixed up these bugs, that's fine. Your TCIs will need to be rebuilt again, though. Just send an email to info@criticker.com and let us know.

BadCosmonaut
Posts: 354
4388 Ratings
Your TCI: na
Joined: Mon Sep 16, 2013 7:08 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by BadCosmonaut »

One thing I didn't like about the implementation of the tier system was that the system would not force at least 10 different tiers for users who use at least 10 different ranks. For example, let's say I rate movies one of the following ratings: 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. That's 10 total. It was possible (depending on how many movies were in each rating) to just have 7 tiers. I wrote about it here and in a few other threads before. I could never understand the reasoning for not forcing 10 tiers (for users with at least 10 different ratings).

Does this new percentile system suffer from that same problem? Like I just made the switch to percentiles. Behind the scenes, are my scores still divided up into any kind of tier system that forces them into just 7 tiers? Or are they split up basically into 101 tiers now (one for each whole number percentile including zero)?

mpowell
Posts: 3831
1200 Ratings
Your TCI: na
Joined: Fri Sep 09, 2005 10:22 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by mpowell »

BadCosmonaut wrote:One thing I didn't like about the implementation of the tier system was that the system would not force at least 10 different tiers for users who use at least 10 different ranks. For example, let's say I rate movies one of the following ratings: 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. That's 10 total. It was possible (depending on how many movies were in each rating) to just have 7 tiers. I wrote about it here and in a few other threads before. I could never understand the reasoning for not forcing 10 tiers (for users with at least 10 different ratings).

Does this new percentile system suffer from that same problem? Like I just made the switch to percentiles. Behind the scenes, are my scores still divided up into any kind of tier system that forces them into just 7 tiers? Or are they split up basically into 101 tiers now (one for each whole number percentile including zero)?


They're now basically split into 100 "tiers" (0-99), with no grouping. A 62 percentile is considered slightly different to a 63 percentile. For this reason, the resulting PSIs are slightly more accurate, as there is a finer granularity.

Also, I forgot to mention, we've revamped our "Help & Information" section, to help explain Percentiles, TCIs and PSIs.

BadCosmonaut
Posts: 354
4388 Ratings
Your TCI: na
Joined: Mon Sep 16, 2013 7:08 am

Re: From Tiers to Percentiles: Introducing a Big New Change to Criticker

Post by BadCosmonaut »

Yeah, that's way better then. I'm not surprised it's more accurate.

Post Reply