Optimal 'Films In Common Minimum'

Introduce yourself to the community or chat with other users about whatever is on your mind
tpbradbury
Posts: 7
3862 Ratings
Your TCI: na
Joined: Sun Nov 07, 2021 12:22 pm

Optimal 'Films In Common Minimum'

Post by tpbradbury »

Edited: Your profile page allows you to select a % Films in Common Minimum.
Here's a suggested solution to calculate the best % to make Criticker more accurate, grateful for scrutiny:

For each % minimum: you record 2 results from the Predictive Accuracy page (you only get this as a sponsor):
1) average difference and 2) no. of results stated at bottom of page, i.e. PSIs generated for films you've already rated. At 3% my accuracy page states a 19.0 average difference and 1386 results (from my 1615 rated films). Take 1386 and divide by 19.0, you get 73, let's call this the "Real accuracy score' (RAS). Now change the % minimum, wait till regeneration i.e. a day, and calculate the new RAS. The RAS should max at a certain %. My max RAS is at 26%. For me this is where 1602 results divided by a mean of 20.9 = 77. What do people think? There may currently be a bias on here, that people think low % is good, as low TCIs, but perhaps this is spuriously low and min % needs to be higher for some to hit optimal. See more detail in my post below.
Last edited by tpbradbury on Sun Jan 02, 2022 3:02 pm, edited 9 times in total.

BadCosmonaut
Posts: 355
4399 Ratings
Your TCI: na
Joined: Mon Sep 16, 2013 7:08 am

Re: Optimal 'Films In Common Minimum'

Post by BadCosmonaut »

I just checked mine, and it's set at a whopping 15% which is 564 films in common. I probably need to lower it.

If you're a subscriber, I know there is a chart that tells you how accurate the PSI is for your current ratings. I'm not a subscriber right now so I can't look at it, but going off memory I think it tells you three things:
  • - how many of your ratings are equal to the PSI prediction,
    - how many of your ratings are within 1 rank of (and not equal to) the PSI prediction,
    - how many of your ratings are 2 or more ranks from the PSI prediction.
The last time I was a subscriber, I remember it said that about 1/3 of my PSI's were perfectly accurate (meaning the PSI and my actual score were equal), 1/3 of my PSI's were within 1 rank, and 1/3 of my PSI's were 2 or more ranks.

Your PSI's are fully regenerated every day you're a subscriber. If you subscribed for two months, on day 1 you could set the films in common minimum to 3%, then every day (after regeneration occurs) make a note of how accurate that % was, then increase the films in common minimum % by 1%. After you get to 50%, change it back to whichever % gave the best results.
I've done math experiments, using results from the Prediction Accuracy page.
Edit: I reread your post after I wrote mine. It sounds like maybe you already did what I suggested above.

tpbradbury
Posts: 7
3862 Ratings
Your TCI: na
Joined: Sun Nov 07, 2021 12:22 pm

Re: Optimal 'Films In Common Minimum'

Post by tpbradbury »

Edited: see my posts above and below for solution to calculate optimal minimum

Hi BadCosmonaut, thank you. Edited
Last edited by tpbradbury on Mon Jan 02, 2023 12:58 pm, edited 3 times in total.

AFlickering
Posts: 641
2994 Ratings
Your TCI: na
Joined: Mon Feb 26, 2007 6:15 pm

Re: Optimal 'Films In Common Minimum'

Post by AFlickering »

mine is at 13%. i haven't tried the above method which may be even more accurate, but my method of working this out was to go through my top TCIs with the lowest films in common and check if they seemed compatible. if the percentage is too low you'll find a lot of them won't be, and then you just keep raising the number and rechecking until you're largely happy with the selection. on the flipside, if you have trusted kumpels like i do, you'll want those to be figuring in your TCIs so if some of them are under the threshold you may wish to lower the percentage for that reason. 13% seemed to provide a good balance for me last i checked (and my PSIs are bewilderingly accurate for the most part), although i should probably recheck it sometime given how many films i've added since i last looked.

tpbradbury
Posts: 7
3862 Ratings
Your TCI: na
Joined: Sun Nov 07, 2021 12:22 pm

Re: Optimal 'Films In Common Minimum'

Post by tpbradbury »

Edited: see solution in my post below

[Hi Aflickering, thank you! yes I tried this method]
Last edited by tpbradbury on Fri Dec 31, 2021 12:00 pm, edited 3 times in total.

tpbradbury
Posts: 7
3862 Ratings
Your TCI: na
Joined: Sun Nov 07, 2021 12:22 pm

Re: Optimal 'Films In Common Minimum'

Post by tpbradbury »

I found a method that calculates a plausible optimum, thoughts?:

For each % minimum: you record 2 results from the Predictive Accuracy page you get if you're a sponsor:
1) average (mean) difference and 2) no. of results stated at bottom of page, i.e. total PSIs generated for films you've already rated. I've rated 1615 films. So 3% means 48 film minimum. Accuracy page states a 19.0 average difference and 1386 results (from 1615 films). Take 1386 and divide by 19.0, you get 73, let's call this the "Real accuracy score' (RAS). Now change the % minimum, wait till regeneration e.g. a day, and calculate a new RAS. The RAS should max at a certain %. My max RAS is at 26%. For me this is where 1602 results divided by a mean of 20.9 = 77. 26% feels high?

Explanation: the average (mean) difference is nominally the average prediction error, which ideally would be zero. But the mean by itself is probably not the best indicator of accuracy, as it maybe spuriously low when % minimum is low, as TCIs are based on a low number of common films. By taking into account the no. of PSIs generated for rated films, this may be the best indicator of accuracy. It normalises the prediction pool size based on error. I studied advanced stats years ago and just looked up sampling theory etc, complicated! My method feels dubious as mean = sum of terms/no. of terms and i'm muliplying by no. of terms again. But the gentle, downward parabola shape of the real accuracy function feels plausible. [I used mean squared but that weights low minimums too much.] Grateful for feedback, e.g. can you calculate what 'optimum' you get? Perhaps start at 10%, then 15, 20, until RAS falls and go back to find max RAS. I know I'm taking this seriously, but nice puzzle to crack if it helps people watch films they find fulfilling

[Useful other thread: viewtopic.php?p=5715#p5715]
Last edited by tpbradbury on Sat Jan 01, 2022 11:27 pm, edited 2 times in total.

AFlickering
Posts: 641
2994 Ratings
Your TCI: na
Joined: Mon Feb 26, 2007 6:15 pm

Re: Optimal 'Films In Common Minimum'

Post by AFlickering »

i'm not currently subscribed and i'm also broke as shit but when i subscribe again i'll definitely give this a whirl and try to wrap my brain around it, thanks!

BadCosmonaut
Posts: 355
4399 Ratings
Your TCI: na
Joined: Mon Sep 16, 2013 7:08 am

Re: Optimal 'Films In Common Minimum'

Post by BadCosmonaut »

Hah! I'm in the same boat as AFlickering. I'll give it a try at some point this year (hopefully within the next few months) when I resub, and I'll report back the results. Thanks for the update/info OP!

Risselada
Posts: 24
4402 Ratings
Your TCI: na
Joined: Mon Jan 11, 2010 3:46 am

Re: Optimal 'Films In Common Minimum'

Post by Risselada »

It's a cool idea tpbradbury. Thanks for the suggestion. I think I'll try it out.

tpbradbury
Posts: 7
3862 Ratings
Your TCI: na
Joined: Sun Nov 07, 2021 12:22 pm

Re: Optimal 'Films In Common Minimum'

Post by tpbradbury »

You can see my max at 26% clearly if y-axis is truncated
Max chart truncated axis.png
Max chart truncated axis.png (43.73 KiB) Viewed 3195 times
But truncating is regarded as misleading and I see why.
If we look with y-axis untruncated it suggests real accuracy doesn't change much from 3 to 50%?
Max chart untruncated axis.png
Max chart untruncated axis.png (35.81 KiB) Viewed 3195 times

Post Reply