"weighted average" of AUC etc. ?

[Harri Saarikoski-2]

must ask, can't deduce

explorer outputs a weighted average of the model quality measures as follows:
TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
0.983     0.556      0.971     0.983     0.977      0.7      O
0.444     0.017      0.571     0.444     0.5        0.72     I
0.956     0.529      0.951     0.956     0.953      0.701
   a   b   c   <-- classified as
 169   3   0 |   a = O
   5   4   0 |   b = I

TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
0.994     0.583      0.921     0.994     0.956      0.872    O
0.417     0.006      0.909     0.417     0.571      0.882    I
0.92      0.51       0.919     0.92      0.907      0.874
   a   b   c   <-- classified as
 163   1   0 |   a = O
  14  10   0 |   b = I

how exactly is e.g. AUC (ROC area) averaged?
--
-----------------
Harri M.T. Saarikoski
M.A, PhD graduate student
Helsinki University
Finland

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Mark Hall-9]

On 30/10/09 5:13 AM, Harri Saarikoski wrote:

> must ask, can't deduce
>
> explorer outputs a weighted average of the model quality measures as
> follows:
> TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
> 0.983     0.556      0.971     0.983     0.977      0.7      O
> 0.444     0.017      0.571     0.444     0.5        0.72     I
> 0.956     0.529      0.951     0.956     0.953      0.701
>     a   b   c <-- classified as
>   169   3   0 |   a = O
>     5   4   0 |   b = I
>
> TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
> 0.994     0.583      0.921     0.994     0.956      0.872    O
> 0.417     0.006      0.909     0.417     0.571      0.882    I
> 0.92      0.51       0.919     0.92      0.907      0.874
>     a   b   c <-- classified as
>   163   1   0 |   a = O
>    14  10   0 |   b = I
>
> how exactly is e.g. AUC (ROC area) averaged?

double aucTotal = 0;
     for(int i = 0; i < m_NumClasses; i++) {
       double temp = areaUnderROC(i);
       if (!Utils.isMissingValue(temp)) {
         aucTotal += (temp * classCounts[i]);
       }
     }

     return aucTotal / classCountSum;


Cheers,
Mark.

--
Mark Hall
Senior Developer/Consultant, Pentaho Open Source Business Intelligence
Citadel International, Suite 340, 5950 Hazeltine National Dr., Orlando, FL
32822, USA
[phone] office, [phone] mobile, [phone] fax,
Skype: mark.andrew.hall, Yahoo: mark_andrew_hall
Download the latest release today <http://www.sourceforge.net/projects/pentaho>

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Harri Saarikoski-2]

ok, so it's class prior probability weighted AUC of class based AUCs, right?

is it (weighted avg of AUC) the 'official' measure of AUC defined as "The AUC is equal to the probability that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one" ? in http://en.wikipedia.org/wiki/Receiver_operating_characteristic

I'm quite sure I could figure that out ultimately, but why think if you can ask, for once ;)
thanks, H

2009/10/29 Mark Hall <mhall@...>

On 30/10/09 5:13 AM, Harri Saarikoski wrote:

must ask, can't deduce

explorer outputs a weighted average of the model quality measures as
follows:
TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
0.983     0.556      0.971     0.983     0.977      0.7      O
0.444     0.017      0.571     0.444     0.5        0.72     I
0.956     0.529      0.951     0.956     0.953      0.701
   a   b   c <-- classified as
 169   3   0 |   a = O
   5   4   0 |   b = I

TP Rate   FP Rate   Precision   Recall  F-Measure   ROC Area  Class
0.994     0.583      0.921     0.994     0.956      0.872    O
0.417     0.006      0.909     0.417     0.571      0.882    I
0.92      0.51       0.919     0.92      0.907      0.874
   a   b   c <-- classified as
 163   1   0 |   a = O
  14  10   0 |   b = I

how exactly is e.g. AUC (ROC area) averaged?

double aucTotal = 0;
   for(int i = 0; i < m_NumClasses; i++) {
     double temp = areaUnderROC(i);
     if (!Utils.isMissingValue(temp)) {
       aucTotal += (temp * classCounts[i]);
     }
   }

   return aucTotal / classCountSum;


Cheers,
Mark.

--
Mark Hall
Senior Developer/Consultant, Pentaho Open Source Business Intelligence
Citadel International, Suite 340, 5950 Hazeltine National Dr., Orlando, FL 32822, USA
[phone] office, [phone] mobile, [phone] fax,
Skype: mark.andrew.hall, Yahoo: mark_andrew_hall
Download the latest release today <http://www.sourceforge.net/projects/pentaho>

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

--
-----------------
Harri M.T. Saarikoski
M.A, PhD graduate student
Helsinki University
Finland

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html