[jira] Created: (MAHOUT-196) bounded values for RecommenderEvaluator

bounded values for RecommenderEvaluator
---------------------------------------

                 Key: MAHOUT-196
                 URL: https://issues.apache.org/jira/browse/MAHOUT-196
             Project: Mahout
          Issue Type: Improvement
          Components: Collaborative Filtering
            Reporter: Jens Grivolla
            Priority: Minor


When evaluating a recommender using RMSRecommenderEvaluator (or some others) on e.g. Netflix data, a recommender gets heavily penalized for predicting values below 1 or above 5 (that are known to be out of the permitted bounds).

It seems to me that it makes no sense to change the recommender to avoid those predictions, since an estimated 6 probably has a greater chance to be highly rated than a predicted 5.1.  I therefore propose to allow truncating predictions to those "legal" values directly in the evaluator and leave the recommenders unchanged, since it is more of a post-processing step than part of the recommender itself.

I added those boundaries to the constructor of RMSRecommenderEvaluator and limit estimatedPreference to the allowed range before calculating "realPref.getValue() - estimatedPreference" and seem to get slightly better scores.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/jira/browse/MAHOUT-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12774025#action_12774025 ]

Sean Owen commented on MAHOUT-196:
----------------------------------

It's an interesting question, yeah. One approach would be to cap this in the recommender, which makes some sense. Why would I ever estimate a movie was rated 6 stars? the only catch is then you lose some ordering information that the estimates provide. A 5.5 star movie should still be recommended before 5.4.

Let me think about a way to incorporate this. I imagine it is indeed just a matter of exposing some way to express limits.

> bounded values for RecommenderEvaluator
> ---------------------------------------
>
>                 Key: MAHOUT-196
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-196
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>            Reporter: Jens Grivolla
>            Priority: Minor
>
> When evaluating a recommender using RMSRecommenderEvaluator (or some others) on e.g. Netflix data, a recommender gets heavily penalized for predicting values below 1 or above 5 (that are known to be out of the permitted bounds).
> It seems to me that it makes no sense to change the recommender to avoid those predictions, since an estimated 6 probably has a greater chance to be highly rated than a predicted 5.1.  I therefore propose to allow truncating predictions to those "legal" values directly in the evaluator and leave the recommenders unchanged, since it is more of a post-processing step than part of the recommender itself.
> I added those boundaries to the constructor of RMSRecommenderEvaluator and limit estimatedPreference to the allowed range before calculating "realPref.getValue() - estimatedPreference" and seem to get slightly better scores.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/jira/browse/MAHOUT-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved MAHOUT-196.
------------------------------

       Resolution: Fixed
    Fix Version/s: 0.3
         Assignee: Sean Owen

I committed a variant on your suggestion. Since these are optional parameters to the object, I implemented them as bean properties rather than constructor args (which I use for "necessary" parameters, though admittedly inconsistently). You can now set a max/min preference value for use in the evaluation.

> bounded values for RecommenderEvaluator
> ---------------------------------------
>
>                 Key: MAHOUT-196
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-196
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Collaborative Filtering
>            Reporter: Jens Grivolla
>            Assignee: Sean Owen
>            Priority: Minor
>             Fix For: 0.3
>
>
> When evaluating a recommender using RMSRecommenderEvaluator (or some others) on e.g. Netflix data, a recommender gets heavily penalized for predicting values below 1 or above 5 (that are known to be out of the permitted bounds).
> It seems to me that it makes no sense to change the recommender to avoid those predictions, since an estimated 6 probably has a greater chance to be highly rated than a predicted 5.1.  I therefore propose to allow truncating predictions to those "legal" values directly in the evaluator and leave the recommenders unchanged, since it is more of a post-processing step than part of the recommender itself.
> I added those boundaries to the constructor of RMSRecommenderEvaluator and limit estimatedPreference to the allowed range before calculating "realPref.getValue() - estimatedPreference" and seem to get slightly better scores.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.