[new] high-precision normalized score

Ideas to improve Criticker and new feature requests, as well as announcements about new enhancements.
livelove
Posts: 759
67 Ratings
Your TCI: na
Joined: Thu Sep 17, 2015 9:36 pm

[new] high-precision normalized score

Post by livelove »

@mpowell:

:idea: I have a suggestion: "high-precision normalized score" (HPNS).

In mathematical terms:

Code: Select all

instead of:
percentile = floor((num_lower + (num_matching / 2) / total_rankings) * 100)

just don't round down:
HPNS = (num_lower + num_matching / 2) / total_rankings * 100
(besides, I don't think your brackets are correctly placed)

I think I have filed this as a feature request a few years ago, but cannot retrieve the posting.
Already back then, when the tier system was still in place, I thought the tiers could be shown to the user (as an easy to grasp visual concept), but are an unnecessary limitation on the back-end, because they are not needed for internal calculations.

      What I mean is, you could have displayed a user's film A with a score of 32 and film B with a score of 39 in the same tier 3 for us users, but don't throw away the decimals – instead keep them for internal calculations: film A is internally stored as (say) "tier" 3.034674716 and film B as 3.289030954

The percentile concept does improve the granularity, as you note yourself, and so is a big step forward. But why don't you go all the way and store a high-precision value (with 10+ decimal places), instead of rounding to the second (tier system) of first (percentile system) pre-decimal point position? Why rounding at all, instead of using a HPNS – at least for internal calculations ?

This would have 2 benefits.

• I'll start with the more important one:

By definition, the tiers introduce rounding errors.
The percentiles introduce only 1 tenth of the tier system's rounding errors (as the precision is higher by a factor of 10).
Why not using even higher precision internally ?

trippingly wrote:Imagine two users see the same 100 films and have the exact same opinion of them.
[…]
The two users still have exactly the same opinions of the films. The only difference is the level of granularity of the two rating systems. How different would the two sets of ratings appear to Criticker?

Using the "midpoint" formula, there would be absolutely no difference if not for rounding. But rounding creates a small difference of 0.5 per film, for a total difference of 50.
Using the "lower bound" formula, the difference is 19 times greater than that: 950.

User trippingly refers to rounding errors in a slightly different context (midpoint vs floor formula), but I am trying to make a similar argument here:

If you are using 10 tiers (your old tier system), a film could be in tier 8 of user A and in tier 9 of user B and thus have a "distance" of 1 tier (corresponding to 10 percentile points), when in fact they are much closer to each other, because the film is in the upper half of user A's tier 8 and in the lower half of user B's tier 9. Example given:

Code: Select all

            user A's tier 8:                         user B's tier 9:
[70,70,72,73,75,77,77,78,79,79,*79*,80]   [*90*,90,90,91,93,95,96,96,98,99]
                                 ╚══════════╝
(side note: I used 79 and 90 as rankings here, but the nominal values don't matter. Users A and B could both give the film the same score of 90 and still have it ranked several tiers away from each other, e.g. tier 6 for user A and tier 10 for user B).

Now with 100 tiers (percentile system) instead of only 10 tiers (old tier system), the rounding error is substantially reduced. But is still there! Even if it's small, why not get rid of it altogether?

Instead of 100 tiers (percentile system), you could use 1000 tiers.
Or 10000 tiers.
Or 100000 tiers.
Every time, the rounding error gets smaller and smaller.

Hence my suggestion of storing a high-precision value (HPNS).

You could either keep the visual output to the user as-is or show higher-precision values or a mix of both (I would display rounded values in the film's rating list, as it is currently done, but possibly display 1 or 2 decimal places at other places, where only 1 value is shown and not a whole list of them).
But regardless of the visual output, calculating with higher precision would reduce the rounding errors and thus increase the prediction accuracy (prediction accuracy index), which you observed yourself:

mpowell wrote:We tested the new system against a wide variety of users, and in nearly every case, accuracy improved

Further increasing the precision/granularity would further increase the accuracy (of both TCIs and PSIs).
(Probably not by an awful lot, but my point is, that the loss off accuracy introduced by the rounding errors are completely unnecessary – and incidentally can get rid of very easily by just not rounding. Calculation speeds of your computers are not even affected. If anything, they are rather sped up, since you omit the rounding.)


• It also has a side-benefit:

www.criticker.com/explain wrote:it's impossible to acheive a percentile rank of 100, because a score can never be lower than itself, and we use the floor as opposed to rounding the number. Percentiles can therefore range from 0 to 99.

trippingly wrote:
AFlickering wrote:
ramynoodle wrote:The movies I gave a score of 100 to should be in the 100th percentile (≈99.67%), but it says 99th instead.

99th is the highest possible percentile because the site is calling the lowest percentile the '0th percentile'. to have percentile called 100th we'd need to get rid of the 0th (just like we had no 0th tier in the old system and went 1-10). might be a good idea, as the current way seems to be confusing people and 1-100 just generally makes more sense than 0-99 IMO, plus it's more in keeping with the previous system.

Yeah, as AFlickering says, this is all due to the fact that if you had both 0 and 100, you would end up with 101 percentiles. And if you rounded in the traditional way, the 0 and 100 "bins" would only be half as big as the other 99 bins (0 would represent the half a percent from 0.00 to 0.49, but 1 would represent the whole percent from 0.50 to 1.49, and so on).
So to get 100 bins of equal size, it seems like percentile systems usually choose to round all decimals either up or down.
A point in favor of the current 0-99 approach is that what's in front of the decimal stays the same, so 99.01% and 99.99% are both counted as 99%. I guess I slightly prefer 0-99 for that reason, but I can totally understand how 1-100 could seem more intuitive too. Either way works. And probably either way, there will still be questions from time to time...

Getting rid alltogether of any kind of "bins" (as trippingly calls them) – both tiers and percentiles – and using a high-precision value instead would get rid of the quoted problem:
User ramynoodle's says "The movies I gave a score of 100 to should be in the 100th percentile (≈99.67%), but it says 99th instead." If you calculate (internally) with a high-precision value, you could display 99.7% or 100% in this case, which solves the problem. ;)

Post Reply