From Tiers to Percentiles: Introducing a Big New Change to Criticker

[mpowell]

when ranking a movie it no longer says "not good" "good" etc based on the tier

I haven't been able to reproduce this. Has anyone else had this problem? Is it only appearing on a certain page?

[trippingly]

Thought I'd also illustrate with my own scores. I only use six... I imagine most Criticker users like a little more granularity in their ratings, but I like not having to make fine distinctions between movies I liked about the same. Anyway, you can see the distribution below...



My old tiers were 1 2 4 6 8 10, which matched this distribution quite well.

You can see, for example, that for me a score of 80 falls close to the middle of the distribution. Using the "midpoint" formula that I recommended above, an 80 for me works out to the 52nd percentile. If you instead use the "lower bound" formula, an 80 works out to the 39th percentile. To my mind, that doesn't capture where my scores of 80 really fall in the overall range.

My top score of 100 works out to the 92nd percentile using the midpoint formula but only the 85th percentile using the lower bound formula. That means that at the moment, when I compare ratings with another user, a film that we both regard as a favorite might differ by as much as 14% (my 85% compared to their 99%).

Now, it totally makes sense that if one user makes fine distinctions and another user doesn't, you will end up with gaps. But using the midpoint formula, everything still lines up right. If I give a score of 100 to the top 15% of the films I've rated, then the percentile rank for that score falls right in the middle of that range at 92%. Whereas using the "lower bound" formula, the fewer distinctions a user makes, the more all of their percentiles will be skewed toward zero. I imagine that as a result, the prediction system will see artificial distance between users who have very different levels of granularity... distance that has nothing to do with their taste in film, just their preferred rating system.

Of course, one remedy is for users to be more granular in their ratings. But one of the things that's so cool about Criticker is that you can use whatever rating system suits you, and it still works great.

Incidentally, I notice in the new information page that for percentiles to work, a user has to use at least five different scores. With the midpoint formula, there's no need for such a minimum.

Let's say there's a user who likes to rate movies as either thumbs up or thumbs down. Their ratings might look this:

0 0 0 0 100 100 100 100 100 100

With the lower bound formula, this user's rating of 100 works out to the 40th percentile. It makes sense not to compare them with other users, because their percentiles are skewed downward.

But with the midpoint formula, this user's 0 works out to the 20th percentile and their 100 works out to the 70th percentile. Still not very granular of course, but reflective of the distribution. I imagine that this user would not end up very high on others' TCI lists, but I also imagine that from their point of view, Criticker could still do a good of calculating and ranking their TCIs and making accurate predictions.

I know that at least one person on Criticker only uses one rating. The midpoint formula would even work for them, sensibly resulting in 50%. Of course, it wouldn't make sense to calculate TCIs and PSIs in that case.

Anyway, hope this helps illustrate why I think this way of calculating percentiles will give more intuitive and more accurate results.

[zevel6]

when ranking a movie it no longer says "not good" "good" etc based on the tier

I haven't been able to reproduce this. Has anyone else had this problem? Is it only appearing on a certain page?

It seems to be OK now - so you can consider this bug fixed However, I noticed that the option to adjust quips no longer works?

I think all the other bugs I mentioned in my prior message still require fixing. I noticed that Zarkon also mentioned the bug regarding percentiles not showing for all titles. The extractions do not work yet too (under any format).

I also want to third the request to add visual demarcations for percentiles in the All Rankings page.

Many thanks for all your efforts - the site is terrific and your support is always appreciated

[trippingly]

One last example that I think is the clearest one yet and then I promise I'll stop spamming the thread.

Imagine two users see the same 100 films and have the exact same opinion of them. Each decides to rank the films in order of preference, and the rankings they come up with are identical.

So the first user signs up for Criticker and gives a score of 1 to the film they liked least, a score of 2 to the film they liked second least, and so on up to 100.

The second user also signs up for Criticker but decides that making these fine distinctions was too much trouble. Going forward, they would like to use a simpler rating system with only five scores, 1 to 5, and so they decide to use this simpler rating system for their first hundred films as well. The bottom twenty films get a 1, the next twenty films get a 2, and so on up to 5.

The two users still have exactly the same opinions of the films. The only difference is the level of granularity of the two rating systems.

How different would the two sets of ratings appear to Criticker?

Using the "midpoint" formula, there would be absolutely no difference if not for rounding. But rounding creates a small difference of 0.5 per film, for a total difference of 50.

Using the "lower bound" formula, the difference is 19 times greater than that: 950.

That's the important takeaway. Below is all the math for anyone who's interested, totally skippable for everyone else.

—

I'll use the midpoint formula first:

(# of films with a lower score + half the # of films with the score in question) / total # of films ranked

That number is then multiplied by 100 and rounded down.

Since in this example the total # of films ranked is 100, we would divide by 100 and then multiply by 100, so we can just skip both those operations.

So it becomes: # of films with a lower score + half the # of films with the score in question, which is then rounded down.

For the first user, who uses a different score for each film...

score 1: 0 + 0.5 = 0.5 ~= 0
score 2: 1 + 0.5 = 1.5 ~= 1
score 3: 2 + 0.5 = 2.5 ~= 2
...
score 100: 99 + 0.5 = 99.5 ~= 99

So for the first user, after rounding down, the percentile is always the score - 1.

For the second user, who uses five scores for twenty films each...

score 1: 0 + 10 = 10
score 2: 20 + 10 = 30
score 3: 40 + 10 = 50
score 4: 60 + 10 = 70
score 5: 80 + 10 = 90

Now to compare each film:

film 1: 10 (user 2) - 0 (user 1) = 10
film 2: 10 - 1 = 9
film 3: 10 - 2 = 8
...
film 10: 10 - 9 = 1
film 11: 10 - 10 = 0
film 12: 10 - 11 = -1
...
film 19: 10 - 18 = -8
film 20: 10 - 19 = -9

When you add up the first block of 20 films, the total difference is 10. And it's the same for the other four blocks, for a total difference of 50.

Now for the lower bound formula:

# of films with a lower score / total # of films ranked

That number is then multiplied by 100 and rounded down.

Here too, we would divide by 100 then multiply by 100, so we can skip both those operations.

That leaves us with just: # of films with a lower score, rounded down.

For the first user...

score 1: 0
score 2: 1
...
score 100: 99

In other words, it's the same as before.

For the second user...

score 1: 0
score 2: 20
score 3: 40
score 4: 60
score 5: 80

Now to compare each film:

film 1: 0 (user 2) - 0 (user 1) = 0
film 2: 0 - 1 = -1
film 3: 0 - 2 = -2
...
film 19: 0 - 18 = -18
film 20: 0 - 19 = -19
film 21: 20 - 20 = 0
film 22: 20 - 21 = -1
film 23: 20 - 22 = -2
...
film 99: 80 - 98 = -18
film 100: 80 - 99 = -19

For each block of twenty films, the total difference is -190, for a grand total difference of -950.

OK, no more spamming the thread, I promise.

[mpowell]

Thought I'd also illustrate with my own scores. I only use six... I imagine most Criticker users like a little more granularity in their ratings, but I like not having to make fine distinctions between movies I liked about the same. Anyway, you can see the distribution below...

Thanks for this detailed breakdown. Actually, now that we have seen this in action, we're already planning on doing something similar. Some users who only rank up to 10, but then have a lot of "10s"... and the resulting percentile "feels" too low.

I'm adding your notes and example into our enhancement report. This is going to be something we get to sooner, rather than later.

[trippingly]

Thanks for this detailed breakdown. Actually, now that we have seen this in action, we're already planning on doing something similar. Some users who only rank up to 10, but then have a lot of "10s"... and the resulting percentile "feels" too low.

I'm adding your notes and example into our enhancement report. This is going to be something we get to sooner, rather than later.

Ah, wonderful, very happy to hear that! (I'm sure I went into more detail than necessary, but interesting thinking about how the recommendation engine works...)

[trippingly]

Also, I think it's really cool that the process Criticker uses to arrive at recommendations is so transparent and clearly explained!

[Suda Balik]

If we do switch, will it be possible to switch back if we don't like it?

"Yes, you'll be able to switch back, if you really want to. But, your TCIs and PSIs will have to be reset again, and this time they won't be fully regenerated. Because of the strain on the server, we don't want to get into a situation where users are switching back-and-forth. Of course, if you're a sponsor of the site, switching back and forth is less of a problem, since full, daily generations of PSIs"

How can I switch back to tiers? My TCI list is always changing and not accurate.
Also I gave scores only divisible to 5 and as a result my max percentile is 93. Tier system seems to be more effective for me.

[zevel6]

If we do switch, will it be possible to switch back if we don't like it?

"Yes, you'll be able to switch back, if you really want to. But, your TCIs and PSIs will have to be reset again, and this time they won't be fully regenerated. Because of the strain on the server, we don't want to get into a situation where users are switching back-and-forth. Of course, if you're a sponsor of the site, switching back and forth is less of a problem, since full, daily generations of PSIs"

How can I switch back to tiers? My TCI list is always changing and not accurate.
Also I gave scores only divisible to 5 and as a result my max percentile is 93. Tier system seems to be more effective for me.

I agree that there does seem to be an issue with the TCI list after changing to percentiles.
For example, my top 12 TCIs currently show a TCI score of under 3 yet when i click the users the TCI shows 100x the number from the TCI list. It's as if the TCI list still calculates it per tier on some users.

[djross]

x