My PSIs vs Actual Score Comparison

Introduce yourself to the community or chat with other users about whatever is on your mind
cameron326
Posts: 153
Your TCI: na
Joined: Fri Feb 25, 2011 8:54 am

My PSIs vs Actual Score Comparison

Post by cameron326 »

Had a spare half hour so decided to make a quick study of the accuracy of my predicted score (PSI) vs actual scores.

This is my method:
I decided to make two separate comparisons - one comparing actual scores vs PSI of "new" films and one comparing "old" films.

I had a hunch that PSI scores for newer films tended to be more accurate for me but i wanted to test this and see just how accurate they were. As I was born in the 1981, I decided that 'new' films for me should consitute films I might have seen when they were actually released, hence the 1990 - 2011 border for new films. Old films I classed as anything before 1990.

I wanted to get a 'random' selection of films for the two surveys, hopefully including a bunch of good bad and average films in each list. So I went to my films, selected 1990 - 2011 and then sorted alphabetically. I then chose the first film to appear under each letter from the 1990 - 2011 films I;d watched, eg the first 'A' film the first 'B' film, etc for inclusion in the list. So I was aiming for 26 random films for each list - one for each letter of the alphabet - but I decided to end up with 21 for each list (as I hadn't seen any old Q, Y, Z films etc).
Here are my results:

Older Films 1920 – 1989

(My Score First - Then PSI)

Airplane! 72 78
Back to the future 90 80
Carrie 78 77
Dead Poet’s Society 73 76
ET 90 83
First Blood 82 76
Gandhi 75 78
Howard the Duck 68 41
Indiana Jones Last Crusade 82 78
Jaws 92 85
Karate Kid 74 72
The Last Empereor 68 75
The Maltese falcon 67 79
North by Northwest 66 83
The Omen 76 74
Pet Semetary 70 70
Raiders of the Lost Ark 86 86
Shadow of a Doubt 87 79
Teen Wolf 70 67
Vertigo 78 83
War of the Roses 84 73

Total Films 21

Number of PSIs within 3 points of actual score (8) 38%
Number of PSIs within 5 points of actual score: (10) 48%
Number of PSIs within 10 points of actual score: (17) 81%
_________________________________________________________________________

Newer Films 1990 - 2011

AI 68 72
Babe 73 73
Cabin Fever 76 68
Dan In real life 50 70
An Education 70 76
Face Off 76 73
Gang of New York 75 74
Hachiko 78 75
I am Legend 72 69
Jackass 73 69
Karate Kid (2009) 62 69
Last Airbender 15 59
Machete 62 73
Nanny Mcphee 70 71
Ocean’s Eleven 75 73
Painted veil 76 75
Ratatouille 83 81
Saving Private Ryan 83 82
Taken 80 75
Unbreakable 69 70
Vacancy 67 63

Total Films 21

Number of PSIs within 3 points of actual score (10) 48%
Number of PSIs within 5 points of actual score: (15) 71%
Number of PSIs within 10 points of actual score: (18) 86%

___________________________________________________________

I think the results speak for themselves. PSI score predictions are very impressive for both newer and older films, although my "newer" film PSIs are more accurate. 71.4% of newer films within 5 points of PSI is an especially impressive statistic.

Both older films and newer films gave a PSI within 10 points for more than 80% of films ranked

Would be interested to hear others statistics and theories.

cameron326
Posts: 153
Your TCI: na
Joined: Fri Feb 25, 2011 8:54 am

Re: My PSIs vs Actual Score Comparison

Post by cameron326 »

I see what you're saying. But I think what you point out only matters in extreme cases - basically just the worst of the worst movies. For example, I gave Last Airbender a '15' out of 100. So of course, its extremely unlikely that the PSI will give such a low rating, even though it might be able to suggest a PSI of Tier 1 or Tier 2.

However, with fairly low, middle and high ranking films, the PSI score works very accurately in many cases.

In the majority of cases the PSI was able to get within 5 points of my actual score. And within 10 points on more than 80% of occasions, so that's pretty impressive.

Comparing tiers though as you say might well show even greater accuracy.

edkrak
Posts: 704
Your TCI: na
Joined: Fri Jun 20, 2008 7:49 am

Re: My PSIs vs Actual Score Comparison

Post by edkrak »

I was curious and made the same thing. I also remember saying once that Criticker is rather useless when it comes to horrors, so I've made the same thing with them also (regardless of year). I'm too lazy to copy all the titles, so I'll write results only.

Older Films 1895 – 1989

Number of PSIs within 3 points of actual score (4) 14,8%
Number of PSIs within 5 points of actual score (8) 29,6%
Number of PSIs within 10 points of actual score: (15) 55,6%
Number of PSIs within 15 points of actual score: (19) 70,4%

Newer Films 1990 - 2011

Number of PSIs within 3 points of actual score (4) 14,8%
Number of PSIs within 5 points of actual score (13) 48,1%
Number of PSIs within 10 points of actual score: (17) 63,0%
Number of PSIs within 15 points of actual score: (20) 74,1%

Horrors

Number of PSIs within 3 points of actual score (1) 3,7%
Number of PSIs within 5 points of actual score (6) 22,2%
Number of PSIs within 10 points of actual score: (9) 33,3%
Number of PSIs within 15 points of actual score: (16) 59,3%

So all my predictions are much worse than yours, but similarly criticker for me predicts rating of older films slightly worse than new ones. As I predicted Criticker is useless for me when predicting horrors. I guess there are probably not enough users here loving cheap zombie/gore/monster films.
Last edited by edkrak on Mon Apr 11, 2011 11:18 am, edited 1 time in total.

cameron326
Posts: 153
Your TCI: na
Joined: Fri Feb 25, 2011 8:54 am

Re: My PSIs vs Actual Score Comparison

Post by cameron326 »

@edkrak,

Good point. Horror films are quite an acquired taste. Many critics seldom if ever give many horror films glowing reviews. I guess what this means is that, just becuase person X and person Y share the same preferences in drama, scifi, thriller, that doesn't necessarily mean that they'll appreciate the same romance or horror films.

Maybe in the future, it might be possible to request PSIs by genre. Therefore TCI information could be generated for specific genres. So although I'm a close match with these TCIs in Drama and Scifi, I use ten other TCIs when seeking horror recommendations. Tricky, and probably more time consuming to be worth it, but just a thought.

TheDenizen
Posts: 1639
Your TCI: na
Joined: Thu Apr 13, 2006 12:51 pm

Re: My PSIs vs Actual Score Comparison

Post by TheDenizen »

edkrak wrote:As I predicted Criticker is useless for me when predicting horrors. I guess there are probably not enough users here loving cheap zombie/gore/monster films.

I'm rating as many as I can! ;)

You've piqued my interest now, I need to compile this data using my own rankings.

cameron326
Posts: 153
Your TCI: na
Joined: Fri Feb 25, 2011 8:54 am

Re: My PSIs vs Actual Score Comparison

Post by cameron326 »

Ok, I see what you're saying now. You're right, comparing Tier to Tier would work better in showing accuracy or inaccuracy of PSI. For example I have a '78' score that is Tier 8 and a '68' score that is Tier 3. So, in fact if a PSI is 10 points off that could be a very big deal (although less so at the top and bottom where my Tiers are much 'wider').

If anyone else can be bothered, they should perhaps compare how many Tiers difference between PSI and actual score. Unfortuantely, it only says what your PSI was not what your predicted Tier was, so its a bit more of a bastard to work out!

TheDenizen
Posts: 1639
Your TCI: na
Joined: Thu Apr 13, 2006 12:51 pm

Re: My PSIs vs Actual Score Comparison

Post by TheDenizen »

I'm gonna stick with PSI as opposed to predicted Tier, but my Tiers are fairly evenly distributed. Not sure how you end up with a mere 10 points representing a range of 5 tiers, to be honest.

OK, I'm using 27 films, one for each letter and one for titles beginning with a number. In cases where the first film listed alphabetically did not have a PSI score (due to not having been rated by enough people), I went with the next film on the list.

Pre 1990
Number of PSIs within 3 points of actual score: 6/27 (22.2%)
Number of PSIs within 5 points of actual score: 12/27 (44.4%)
Number of PSIs within 10 points of actual score: 18/27 (66.7%)
Number of PSIs within 15 points of actual score: 22/27 (81.5%)
Number of PSIs more than 30 points off of actual score: 1/27 (3.7%)

Post 1990
Number of PSIs within 3 points of actual score: 5/27 (18.5%)
Number of PSIs within 5 points of actual score: 12/27 (44.4%)
Number of PSIs within 10 points of actual score: 18/27 (66.7%)
Number of PSIs within 15 points of actual score: 22/27 (81.5%)
Number of PSIs more than 30 points off of actual score: 0/27 (0%)

So all in all, a pretty much identical set of stats for pre and post 1990...not to mention a staggeringly high compatibility with PSI scores. Having over 80% within 15 points is pretty remarkable IMO.

For interests sake, I also did a couple of comparisons based on genre. Horror to compare with edkrak, and comedy since the handful of films that I noticed differed most drastically from the PSIs were comedies.

Horror
Number of PSIs within 3 points of actual score: 5/27 (18.5%)
Number of PSIs within 5 points of actual score: 11/27 (40.7%)
Number of PSIs within 10 points of actual score: 17/27 (63%)
Number of PSIs within 15 points of actual score: 17/27 (63%)
Number of PSIs more than 30 points off of actual score: 5/27 (18.5%)

Like edkrak, PSIs for my horror movie scores aren't quite as accurate as for other genres, but they're pretty decent. When they miss, they tend to miss by a lot though.

Comedy
Number of PSIs within 3 points of actual score: 4/27 (14.8%)
Number of PSIs within 5 points of actual score: 7/27 (25.9%)
Number of PSIs within 10 points of actual score: 12/27 (44.4%)
Number of PSIs within 15 points of actual score: 14/27 (51.9%)
Number of PSIs more than 30 points off of actual score: 3/27 (11.1%)

Not sure what this set of numbers says other than I don't really like a lot of comedies that most people love. There were another 4 or 5 films that were very close to being off by 30 points. Interesting to compare barely over 50% within 15 points of the PSI compared to over 80% for all film genres together.

You could also argue that the sample size is too small to draw any significant conclusions from...but I ain't doing this calculation for over 2000 flicks. 8-)

PeaceAnarchy
Posts: 654
Your TCI: na
Joined: Tue Dec 12, 2006 1:47 am

Re: My PSIs vs Actual Score Comparison

Post by PeaceAnarchy »

I did it with tiers. One thing I noticed is that most of the ones that are really far off are films Criticker overrates, rather than films I love against Criticker's expectation. I agree that the sample size is pretty small but I think it's enough to conclude that criticker is pretty impressive.

Pre 1990
7 no dif
3 by 1
7 by 2
6 by 3
2 by 4
1 each by 6 and 7

So 37% within one tier
63% within 2
85% within 3

Post 1990
5 no dif
10 by 1
5 by 2
5 by 3
2 by 4

So 56% within one tier
74% within 2
93% within 3

edkrak
Posts: 704
Your TCI: na
Joined: Fri Jun 20, 2008 7:49 am

Re: My PSIs vs Actual Score Comparison

Post by edkrak »

I guess that one of the reasons why horror predictions suck for me is because people here very rarely score horrors in their highest tiers and I do it often. I hardly have any horror PSIs of 85 and higher and even them are mostly borderline horrors.

TheDenizen
Posts: 1639
Your TCI: na
Joined: Thu Apr 13, 2006 12:51 pm

Re: My PSIs vs Actual Score Comparison

Post by TheDenizen »

edkrak, it's our responsibility to bring up the average scores for these films.

Italian cannibal/gore/zombie flicks are some of my faves. :D

Post Reply