WEKA Learning Curve, amount of training instances Bug

[wessel van persie]

Dear All,

This tutorial shows you how to generate a Learning curve using the WEKA experimenter.
http://weka.wikispaces.com/Learning+curves
I followed this tutorial exactly.

Strangely enough when looking at the experiment results the amount of training instances used did not vary.
Number_of_training_examples, min: 356 max 357.
Since the percentage correct does vary, from 62% to 90% correct,
I think the experimenter is logging the wrong parameter.
It does not record the actual training instances used inside the loop.

Is there a way to fix this?
I uploaded my experiment file here:
http://student.science.uva.nl/~wluijben/term.exp

Regards,

Wessel


_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Peter Reutemann-3]

> This tutorial shows you how to generate a Learning curve using the WEKA
> experimenter.
> http://weka.wikispaces.com/Learning+curves
> I followed this tutorial exactly.
>
> Strangely enough when looking at the experiment results the amount of
> training instances used did not vary.
> Number_of_training_examples, min: 356 max 357.
> Since the percentage correct does vary, from 62% to 90% correct,
> I think the experimenter is logging the wrong parameter.
> It does not record the actual training instances used inside the loop.
>
> Is there a way to fix this?
> I uploaded my experiment file here:
> http://student.science.uva.nl/~wluijben/term.exp

A few things:
- What version of Weka are you using?
- Don't make binary versions of experiments available as they're not
compatible among different versions of Weka. Use the XML format
instead.
- Don't forget to upload your dataset(s) as well. The experiment
doesn't store the dataset(s), only the location.
- Can you reproduce this behavior also with the latest snapshot of
your branch of Weka (book, stable-3.6, developer)?

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[wessel van persie]

Dear Peter,

I am using the latest version of weka: 3.6.1.
I uploaded the experiment xml, dataset, and results.arff here:
http://student.science.uva.nl/~wluijben/weka_learning_curve/

Regards,

Wessel Luijben

On Sun, Oct 25, 2009 at 12:58 AM, Peter Reutemann <fracpete@...> wrote:

> This tutorial shows you how to generate a Learning curve using the WEKA
> experimenter.
> http://weka.wikispaces.com/Learning+curves
> I followed this tutorial exactly.
>
> Strangely enough when looking at the experiment results the amount of
> training instances used did not vary.
> Number_of_training_examples, min: 356 max 357.
> Since the percentage correct does vary, from 62% to 90% correct,
> I think the experimenter is logging the wrong parameter.
> It does not record the actual training instances used inside the loop.
>
> Is there a way to fix this?
> I uploaded my experiment file here:
> http://student.science.uva.nl/~wluijben/term.exp

A few things:
- What version of Weka are you using?
- Don't make binary versions of experiments available as they're not
compatible among different versions of Weka. Use the XML format
instead.
- Don't forget to upload your dataset(s) as well. The experiment
doesn't store the dataset(s), only the location.
- Can you reproduce this behavior also with the latest snapshot of
your branch of Weka (book, stable-3.6, developer)?

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Peter Reutemann-3]

> I am using the latest version of weka: 3.6.1.

Can you reproduce the behavior with the lastest snapshot of stable-3.6?

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[wessel van persie]

Yes, the behaviour with the latest snapshot is the same.
I uploaded another file:
http://student.science.uva.nl/~wluijben/weka_learning_curve/out_with_snapshot.arff

Cheers,

Wessel

On Sun, Oct 25, 2009 at 1:54 AM, Peter Reutemann <fracpete@...> wrote:

> I am using the latest version of weka: 3.6.1.

Can you reproduce the behavior with the lastest snapshot of stable-3.6?

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html

[Peter Reutemann-3]

> Yes, the behaviour with the latest snapshot is the same.
> I uploaded another file:
> http://student.science.uva.nl/~wluijben/weka_learning_curve/out_with_snapshot.arff

I had a look at your setup. You're using the FilteredClassifier
approach and changing the size of the dataset that the base
classifier, in your case J48, gets trained with. The Experimenter
*always* provides the classifiers in the experiment, in your case the
FilteredClassifier, with the same dataset and therefore with the same
amount of training instances. The number of training instances happens
inside the FilteredClassifier and is not visible to the Experimenter
(it only modifies parameter settings).

In a nutshell: not a bug, as the Experimenter just runs several
different classifier setups on the same dataset.

Cheers, Peter
--
Peter Reutemann, Dept. of Computer Science, University of Waikato, NZ
http://www.cs.waikato.ac.nz/~fracpete/           Ph. [phone]

_______________________________________________
Wekalist mailing list
Send posts to: Wekalist@...
List info and subscription status: https://list.scms.waikato.ac.nz/mailman/listinfo/wekalist
List etiquette: http://www.cs.waikato.ac.nz/~ml/weka/mailinglist_etiquette.html