Feature Requests : TCI Regenerating

[Quicky]

Hi,

I have some doubts about how the TCI regenerating works and how often it is done. When I was messing with my '% common movies' setting in my profile, I noticed that Criticker isn't refreshing the TCI's 'live' according to this setting. I understand this is because the load on the server would be too heavy. It only refreshes the TCI for a certain user when you go to his or her page. Can somebody enlighten me as to when exactly the TCI for a user is regenerated?

In either case... As some of you might have read here I have been mathematically 'weighing' the importance of Criticker in my personal recommendations. I recently wrote some code to load all critics pages in order to (re)generate the TCI's 'manually' so to speak (My apologies to the server for the extra load).

And I noticed something striking... Before I generated the TCI's manually, the weight of my Criticker PSI parameter was about 23%, while IMDB was at 29%. After I generated the TCI's, PSI jumped to 34% while IMDB dropped to 18%. What this means is that after I generated the TCI's, my program found that my PSI's became much more valuable than before they were generated.

My point with this post is the following: without the regular and complete TCI (re)generating, Criticker.com is much less valuable and reliable than it could be.

So consider this post another request for manual (weekly?) TCI and PSI regenerating. DO something about it!

[mpowell]

To answer your question: currently Criticker regenerates a set of TCIs every time you rank a new movie, or go on another user's page. Also, every two weeks we do a large set of regenerations for all users.

Plans are underway to regularly schedule full TCI & PSI recalculations (or at least, full recalcs for frequently-active users). The first step was to get faster hardware, which we've done, and the next step is to see how much we can push it, which we've got planned.

Thanks for your confirmation that a fresh set of TCIs & PSIs does improve results! That's good to know.

[Quicky]

To answer your question: currently Criticker regenerates a set of TCIs every time you rank a new movie, or go on another user's page. Also, every two weeks we do a large set of regenerations for all users.

Plans are underway to regularly schedule full TCI & PSI recalculations (or at least, full recalcs for frequently-active users). The first step was to get faster hardware, which we've done, and the next step is to see how much we can push it, which we've got planned.

Thanks for this info. This sounds pretty good.

Thanks for your confirmation that a fresh set of TCIs & PSIs does improve results! That's good to know.

Indeed. The odd thing is that you don't really *think* that the recommendations are better because your TCI's don't *seem* to be very different. My top 10 TCI's were still in the 1.95 - 2.20 range with the % common movies at 36. I think the improvement gain came from all the TCI's *beyond* the top 10.

Another thing I've noticed that surprised me after I regenerated all the TCI's, was that the accuracy of my PSI's went up when I lowered the % of common movies.

At 50% common movies I got: 32% PSI, 23% IMDB
At 30% common movies I got: 34% PSI, 19% IMDB
At 10% common movies I got: 39% PSI, 18% IMDB
Lower % gives higher PSI compared to IMDB.

This surprised me a bit because my top 10 TCI's at 50% have a lot more movies to go on and their taste *seems* to be better because you have a lot more common movies than the TCI's at 10%. It *seemed* as if the 10% TCI's weren't as valuable. But apparently that is not the case. I think the reason for this is that at 50%, Criticker has a lot less data available to use compared to when you use 10%. So probably the PSI's get less accurate, especially for the less common movies because there aren't enough users available to use data from.

Because of this result, I then tested 20% and got something weird: 33% PSI and 23% IMDB. Maybe I made a mistake there, or maybe my numbers aren't as reliable as I thought . I'll have to test that again some time.

So........... what I'm currently thinking, is that your top 10 TCI's are more valuable to the user at 50% (me) when you're casually looking for a good movie, whereas for the core business of PSI generating you really need to lower the % to get the best results.

Now... this makes me wonder... how good are the PSI results for people with 1500 rated movies (unlike my 400)? For them 10% is 150 common movies and 50% is 750 movies, right?. How many users are there on Criticker that have 750 movies common with someone else? Don't the recommendations get worse when you've rated so many movies because you can't lower the % of common movies lower than 10%? If a user with 1500 rated movies sets his % at 50, doesn't he get a 'N/A' TCI for all but a few users?

Maybe you shouldn't demand a % of common movies, but an absolute number of common movies? For me, 10% gave the best results = 40 movies, whereas 50% gave the worst results (200 movies).

Something to think about... If you have some further insights into this, I'd gladly hear it .

[td888]

Now... this makes me wonder... how good are the PSI results for people with 1500 rated movies (unlike my 400)? For them 10% is 150 common movies and 50% is 750 movies, right?. How many users are there on Criticker that have 750 movies common with someone else? Don't the recommendations get worse when you've rated so many movies because you can't lower the % of common movies lower than 10%? If a user with 1500 rated movies sets his % at 50, doesn't he get a 'N/A' TCI for all but a few users?

Maybe you shouldn't demand a % of common movies, but an absolute number of common movies? For me, 10% gave the best results = 40 movies, whereas 50% gave the worst results (200 movies).

Something to think about... If you have some further insights into this, I'd gladly hear it .

I totally agree. I've about 1000+ movies ranked. I noticed the more movies I rank ,the further away my closest TCI's get.... I lowered my % of common movies, which gives me better recommendations, I feel. But it's a bit strange that ranking more movies give me worse recommendations....

[Spunkie]

I'm not sure it works the way you think. Common movies are set to %10 for me, to check I set it to %50 and my 50th kumpel was still the same, I also checked a few movies psi for both, psi's stayed the same. Maybe it checks both sides of the equation, if I have seen the set % of your movies you get listed for me, still I'm confused about that.

[Quicky]

I'm not sure it works the way you think. Common movies are set to %10 for me, to check I set it to %50 and my 50th kumpel was still the same, I also checked a few movies psi for both, psi's stayed the same. Maybe it checks both sides of the equation, if I have seen the set % of your movies you get listed for me, still I'm confused about that.

Are you sure all your TCI's and PSI's were updated after you changed the setting? Just changing your % setting doesn't change all your TCI's and PSI's immediately. You should be able to see this in your top 10 TCI's after going from 10% to 50%. When you're at 10% and all your TCI's are up to date and you then change it to 50%, some TCI's will still be listed in your top 10 even though they don't have enough common movies. You need to either wait for Criticker to update them all every two weeks (?) or you need to update them manually by going to every user page and every movie you want the PSI from.

[UPDATE] For me this means I need to load about 5000 TCI's who have 10% common movies or more. Since I need the PSI's for all the movies I've seen I then need to load all 412 pages of the movies I've seen. Obviously I'm not doing this simply by hand.

[Spunkie]

Yes, you are right, I just tried it on the fly so no change whatsoever took effect .

[Quicky]

I totally agree. I've about 1000+ movies ranked. I noticed the more movies I rank ,the further away my closest TCI's get.... I lowered my % of common movies, which gives me better recommendations, I feel. But it's a bit strange that ranking more movies give me worse recommendations....

It's actually quite unlogical that if you rate more movies, there will be more movies listed with a PSI of '???'. The more movies you've rated, the more common movies your TCI's will need to have. The more common movies your TCI's need to have, the less will be available for PSI calculating and thus the more movies will be listed with a PSI of '???'.

I think this is an important flaw that should be addressed, since Criticker should be encouraging to rate more movies and not the other way around, right?...

[Andemon]

Hrm...
I'm not sure if that automatic updating works at all. I've certainly been here for longer than two weeks, and as far as I've seen, my TCIs haven't been updated yet. At least, I haven't noticed any major changes. Ranking new movies doesn't seem to do anything to my TCIs, either.

The only way that I can force them to update is by viewing random user profiles, which I do frequently.

Also, Criticker seems to "only" keep track of 1000 highest TCIs, any TCIs beyond those are ignored when generating PCIs. It probably made perfect sense back when there were fewer users, but maybe now it should be increased; doubled even? I'd say that it's better to receive somewhat erratic PCIs for rarely watched movies than just '???', though I suppose that some may disagree...

[oddball]

I totally agree. I've about 1000+ movies ranked. I noticed the more movies I rank ,the further away my closest TCI's get.... I lowered my % of common movies, which gives me better recommendations, I feel. But it's a bit strange that ranking more movies give me worse recommendations....

Hahaha,... yeah I've noticed that too, on other sites as well [movielens and blockbuster]. But if you think about it is actually very logical that you get worse recommendations the more movies you rate... I mean all these programs work through trying to match your tastes with the tastes of other users... and the more movies are rated the more opportunity for heavy disagreement ensues. I for example am very picky with movies and some minor details or a certain mood in a movie can totally turn me off of a movie that I know is objectivly solid, if not even great. I'm already afraid of rating some movies that I know I probably should like in order to fit my profile, just because I just fucking hate them. Either that or I give them mild ratings.

Memento for example, I fucking hated that movie [ and I can't stand Nolan in general << the bastard fucked up Batman FFS ... but that would be already my next point]. Sure the plot was clever, the actors where within the realm of capability and it had such a nice twist at the end. But I just fucking hated every fukken minute of it. I got a four point something rec on movielens for it though... WHY?! Well because I actually am a fan of non mainstream, clever, expiremental movies. So if I were to rate Memento like I'd want to it could confuse the system. OTOH hatred or extrem liking of some very disputed movies can have very positive effects. But not on movies that are either commonly loved or hated by the type of people that mostly like the same stuff as you.

Now my next point,... which might be pretty far out there.... and could also lead to some heavy misses. But how about adding actors and directors that you usually either love or hate into the equation? I mean I'll pretty much eat up anything with Christina Ricci in it as much as I can't watch nothing that has Tom Cruise or Liv Taylor in it... and any appearance of Robert Duvall will drastically lower my viewing pleasure[yah,... I'm odd ]! Likewise I'm able to enjoy almost everything that Woody Allen touches to at least a certain degree... OK I could've passed on Scoop, but you get what I mean. Scoop didn't piss me off though, it was just goddamn boring. I could bring on loads more examples but you get the point.