Specifying Folds for Leave One Out Cross Validation

[susan mckeever]

I want to use Weka on a dataset that consists of 26 days of timestamped activity data from a smart home.  I'd like to run "Leave one out" cross validation, where one is a single day's worth of data.   I would then like to  follow that experiment with with 'Leave two days out' cross validation and so on. Does Weka have some way of letting me specify the contents of  cross validation folds to support what I want to do? Or will I have to manually  use Training and Test file options under the Explorer Classify to run each fold of the tests manually?

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Peter Reutemann-3]

> I want to use Weka on a dataset that consists of 26 days of timestamped
> activity data from a smart home.  I'd like to run "Leave one out" cross
> validation, where one is a single day's worth of data.   I would then like
> to  follow that experiment with with 'Leave two days out' cross validation
> and so on. Does Weka have some way of letting me specify the contents of
> cross validation folds to support what I want to do? Or will I have to
> manually  use Training and Test file options under the Explorer Classify to
> run each fold of the tests manually?

I reckon, you will have to use your own evaluation, as Weka doesn't
allow you to pick a certain values for evaluation. The groovy script
"CustomCV.groovy" in the "Downloads" section of the following wiki
article might be helpful:
  http://weka.wikispaces.com/Using+Weka+from+Groovy

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[susan mckeever]

Thanks Peter

On Fri, Nov 6, 2009 at 9:22 PM, Peter Reutemann <fracpete@...> wrote:

> I want to use Weka on a dataset that consists of 26 days of timestamped
> activity data from a smart home.  I'd like to run "Leave one out" cross
> validation, where one is a single day's worth of data.   I would then like
> to  follow that experiment with with 'Leave two days out' cross validation
> and so on. Does Weka have some way of letting me specify the contents of
> cross validation folds to support what I want to do? Or will I have to
> manually  use Training and Test file options under the Explorer Classify to
> run each fold of the tests manually?

I reckon, you will have to use your own evaluation, as Weka doesn't
allow you to pick a certain values for evaluation. The groovy script
"CustomCV.groovy" in the "Downloads" section of the following wiki
article might be helpful:
 http://weka.wikispaces.com/Using+Weka+from+Groovy

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html