[ruby-core:26488] Add Standard Deviation Function to Math Module

[Daniel Cohen-4]

This patch adds a Standard Deviation function to the Math Module. It takes an array and finds the standard deviation of all its elements, as long as they are numbers. Do you think this belongs in the Math or Array Modules?

Thanks,
Daniel Cohen

[Yukihiro Matsumoto]

Hi,

In message "Re: [ruby-core:26488] Add Standard Deviation Function to Math Module"
    on Tue, 3 Nov 2009 04:49:47 +0900, Daniel Cohen <danielc2017@...> writes:

|This patch adds a Standard Deviation function to the Math Module. It takes
|an array and finds the standard deviation of all its elements, as long as
|they are numbers. Do you think this belongs in the Math or Array Modules?

Functions in Math modules are taken from math function in UNIX, at
least on some platforms.  So, adding other function may require some
discussion.   Maybe we need to add such functions to some statistic
related module.

                                                        matz.

[Daniel Cohen-4]

OK,
I was thinking that a Statistics module would be a good addition to Ruby. If you decide that it would be worth creating one, I would be happy to contribute.

Sincerely,
Daniel Cohen

On Mon, Nov 2, 2009 at 8:16 PM, Yukihiro Matsumoto <matz@...> wrote:

Hi,

In message "Re: [ruby-core:26488] Add Standard Deviation Function to Math Module"

   on Tue, 3 Nov 2009 04:49:47 +0900, Daniel Cohen <danielc2017@...> writes:

|This patch adds a Standard Deviation function to the Math Module. It takes
|an array and finds the standard deviation of all its elements, as long as
|they are numbers. Do you think this belongs in the Math or Array Modules?

Functions in Math modules are taken from math function in UNIX, at
least on some platforms.  So, adding other function may require some
discussion.   Maybe we need to add such functions to some statistic
related module.

                                                       matz.

[Nobuyoshi Nakada-2]

Hi,

At Tue, 3 Nov 2009 11:05:36 +0900,
Daniel Cohen wrote in [ruby-core:26490]:
> I was thinking that a Statistics module would be a good addition to Ruby. If
> you decide that it would be worth creating one, I would be happy to
> contribute.

statistics/extconf.rb and statistics/statistics.c.


require 'mkmf'
create_makefile('statistics')

#include <math.h>
#include <ruby.h>

struct stat_stdev_args {
    double sum, sum2;
    long num;
};

static VALUE
stdev_i(VALUE elem, VALUE arg, int argc, VALUE *argv)
{
    struct stat_stdev_args *a = (void *)arg;
    double x = NUM2DBL(elem);
    a->sum += x;
    a->sum2 += x*x;
    a->num++;
    return Qnil;
}

/*
 *  call-seq:
 *     Statistics.stdev(enum)   =>   double
 *
 *  Computes the standard deviation of the elements in an <i>enum</i>
 *  If they are all numbers.
 */
 
static VALUE
stat_stddev(VALUE obj, VALUE list)
{
    long i;
    struct stat_stdev_args args;
    double mean, stddev;

    args.sum = args.sum2 = 0.0;
    args.num = 0;
    rb_iterate(rb_each, list, stdev_i, (VALUE)&args);
    mean = args.sum / args.num;
    stddev = sqrt(args.sum2 / args.num - mean*mean);
    return DBL2NUM(stddev);
}

void
Init_statistics(void)
{
    VALUE mStatistics = rb_define_module("Statistics");
    rb_define_singleton_method(mStatistics, "stddev", stat_stddev, 1);
}


--
Nobu Nakada

[Yukihiro Matsumoto]

Hi,

In message "Re: [ruby-core:26490] Re: Add Standard Deviation Function to Math Module"
    on Tue, 3 Nov 2009 11:05:36 +0900, Daniel Cohen <danielc2017@...> writes:

|I was thinking that a Statistics module would be a good addition to Ruby. If
|you decide that it would be worth creating one, I would be happy to
|contribute.

Although I have no knowledge and experience about statistics, I would
love to support technically.  [ruby-core:26491] would be a good
starting point.

                                                        matz.

[Daniel Cohen-4]

Matz,
Thanks for your support, I'll start working on a Statistics Module based on [ruby-core:26491]

Thanks,
Daniel

On Tue, Nov 3, 2009 at 6:54 AM, Yukihiro Matsumoto <matz@...> wrote:

Hi,

In message "Re: [ruby-core:26490] Re: Add Standard Deviation Function to Math   Module"

   on Tue, 3 Nov 2009 11:05:36 +0900, Daniel Cohen <danielc2017@...> writes:

|I was thinking that a Statistics module would be a good addition to Ruby. If
|you decide that it would be worth creating one, I would be happy to
|contribute.

Although I have no knowledge and experience about statistics, I would
love to support technically.  [ruby-core:26491] would be a good
starting point.

                                                       matz.

[Daniel Cohen-4]

Hi,

I am attaching a preliminary version of the statistics module. It implements both kinds of standard deviation, as well as generating random numbers and finding values for a normal distribution among other things.
 
I am currently working on a feature to perform linear curve fitting or regression on a set of points, however, I am running into trouble on how to sum the products of the x and y. For example, I have two enumerations xlist and ylist. Both are the same length. I want to multiply each pair and then add them to the running total. For example, (x1 * y1) + (x2 * y2) + ... for however many pairs I have. Any suggestions on how to do this cleanly would be much appreciated.

The odd include for ruby.h is specific to my development environment.

Sincerely,
Daniel

On Tue, Nov 3, 2009 at 7:39 AM, Daniel Cohen <danielc2017@...> wrote:

Matz,
Thanks for your support, I'll start working on a Statistics Module based on [ruby-core:26491]

Thanks,
Daniel

On Tue, Nov 3, 2009 at 6:54 AM, Yukihiro Matsumoto <matz@...> wrote:

Hi,

In message "Re: [ruby-core:26490] Re: Add Standard Deviation Function to Math   Module"

   on Tue, 3 Nov 2009 11:05:36 +0900, Daniel Cohen <danielc2017@...> writes:

|I was thinking that a Statistics module would be a good addition to Ruby. If
|you decide that it would be worth creating one, I would be happy to
|contribute.

Although I have no knowledge and experience about statistics, I would
love to support technically.  [ruby-core:26491] would be a good
starting point.

                                                       matz.

[Yusuke ENDOH]

Hi,

2009/11/3 Yukihiro Matsumoto <matz@...>:
> |I was thinking that a Statistics module would be a good addition to Ruby. If
> |you decide that it would be worth creating one, I would be happy to
> |contribute.
>
> Although I have no knowledge and experience about statistics, I would
> love to support technically.  [ruby-core:26491] would be a good
> starting point.


I also think that statistics module will be really useful and important.
But it is arguable for me about how to embed:

- Should it be a core-embedded library, not standard library nor gem?
- Should it be written in C, not Ruby?
- Should it be written from the scratch?  Why don't we import existing
  well-tested library?  For example, NArray provides standard deviation.

I with there were gem.ruby-lang.org :-p

--
Yusuke ENDOH <mame@...>

[Cameron McBride]

Hello,

On Tue, Nov 3, 2009 at 16:30, Daniel Cohen <danielc2017@...> wrote:
> I am attaching a preliminary version of the statistics module. It implements
> both kinds of standard deviation, as well as generating random numbers and
> finding values for a normal distribution among other things.

I have been using ruby with basic statistics for many years now, and
over 90% of the time I use NArray.  While I think this is generally a
good idea, it might be worth considering the bounds of this module.
There are a lot of statistics to implement.  I'm glad you included the
two typical versions of stddev.

ruby's rand function implements MT19937, IIRC.  NArray also uses this.
 The proposed statistics module uses the C rand() I believe.  If so,
it might be worth using the MT algorithm.

For basic statistics, might I suggest things like the median and
quantiles be included?

> I am currently working on a feature to perform linear curve fitting or
> regression on a set of points, however, I am running into trouble on how to
> sum the products of the x and y.

I think this might be beyond the scope of a basic Statistics module.

Anyhow, just some thoughts from a lurker who loves using ruby in
science applications.

Cameron

[Cameron McBride]

On Tue, Nov 3, 2009 at 16:56, Yusuke ENDOH <mame@...> wrote:

> Hi,
>
> 2009/11/3 Yukihiro Matsumoto <matz@...>:
>> |I was thinking that a Statistics module would be a good addition to Ruby. If
>> |you decide that it would be worth creating one, I would be happy to
>> |contribute.
>>
>> Although I have no knowledge and experience about statistics, I would
>> love to support technically.  [ruby-core:26491] would be a good
>> starting point.
>
>
> I also think that statistics module will be really useful and important.
> But it is arguable for me about how to embed:
>
> - Should it be a core-embedded library, not standard library nor gem?
> - Should it be written in C, not Ruby?
> - Should it be written from the scratch?  Why don't we import existing
>  well-tested library?  For example, NArray provides standard deviation.

I still don't understand why NArray isn't in the stdlib.  It'd be
great to "bless" a simple and elegant library like this to build off
of (for specific C extensions, etc).

Cameron

[Joel VanderWerf]

Cameron McBride wrote:
> I still don't understand why NArray isn't in the stdlib.  It'd be
> great to "bless" a simple and elegant library like this to build off
> of (for specific C extensions, etc).

I agree. NArray is worth it for its own sake, not just to answer the
need for a (basic) statistics module. Can NArray even replace matrix.rb
eventually?

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

[James Edward Gray II]

On Nov 3, 2009, at 8:28 PM, Cameron McBride wrote:

> I still don't understand why NArray isn't in the stdlib.  It'd be
> great to "bless" a simple and elegant library like this to build off
> of (for specific C extensions, etc).

Yeah, I've always wished it and RBTree could be added to the standard  
library.

James Edward Gray II

[Joel VanderWerf]

James Edward Gray II wrote:
> On Nov 3, 2009, at 8:28 PM, Cameron McBride wrote:
>
>> I still don't understand why NArray isn't in the stdlib.  It'd be
>> great to "bless" a simple and elegant library like this to build off
>> of (for specific C extensions, etc).
>
> Yeah, I've always wished it and RBTree could be added to the standard
> library.

"bless" you for suggesting that!

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

[Daniel Cohen-4]

Cameron,
Thanks for your suggestions, I'll work on the median and quantiles as well. Now that I think about it, I agree with you that a linear regression function probably doesn't belong in a basic stat module.

Thanks,
Daniel

On Tue, Nov 3, 2009 at 10:08 PM, Joel VanderWerf <vjoel@...> wrote:

James Edward Gray II wrote:

On Nov 3, 2009, at 8:28 PM, Cameron McBride wrote:

I still don't understand why NArray isn't in the stdlib.  It'd be
great to "bless" a simple and elegant library like this to build off
of (for specific C extensions, etc).

Yeah, I've always wished it and RBTree could be added to the standard library.

"bless" you for suggesting that!

--
     vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

[Yugui (Yuki Sonoda)]

2009/11/4 Yusuke ENDOH <mame@...>:
> - Should it be a core-embedded library, not standard library nor gem?
> - Should it be written in C, not Ruby?
> - Should it be written from the scratch?  Why don't we import existing
>  well-tested library?  For example, NArray provides standard deviation.

I believe statistics library is worth to bundled with Ruby.
But, at first, it should be a gem until it will get stable enough and
will be widely used.

BTW, I think NArray satisfies the conditions; stable enough and widely used.

-- Yuki Sonoda (Yugui)

[James Edward Gray II]

On Nov 7, 2009, at 8:06 AM, Yugui wrote:

> BTW, I think NArray satisfies the conditions; stable enough and  
> widely used.

Hooray!  Can we beg for RBTree too?  :)

James Edward Gray II

[NARUSE, Yui-2]

Daniel Cohen wrote:

> I am attaching a preliminary version of the statistics module. It
> implements both kinds of standard deviation, as well as generating
> random numbers and finding values for a normal distribution among other
> things.
>  
> I am currently working on a feature to perform linear curve fitting or
> regression on a set of points, however, I am running into trouble on how
> to sum the products of the x and y. For example, I have two enumerations
> xlist and ylist. Both are the same length. I want to multiply each pair
> and then add them to the running total. For example, (x1 * y1) + (x2 *
> y2) + ... for however many pairs I have. Any suggestions on how to do
> this cleanly would be much appreciated.
>
> The odd include for ruby.h is specific to my development environment.

I'm not a statistics people but some thought about it.

First this argument is enum, but it should receive Typed Array as Float
(or Narray.float(size)) for optimization.
People who use Statistics hate its large number of NUM2DBL
and moreover calling each methods, doesn't they?

If it uses enum as argument, it must call Enumerable#enum.
This means this Statistics library can't run in parallel.

Second is Statistics.rand().
Don't call srand if you want random value.
This implementation allow users to guess the return value by time.

Simply call rb_f_rand() seems good.

And stat_cdf's implementation is broken.

--
NARUSE, Yui  <naruse@...>

[NARUSE, Yui-2]

Yugui wrote:
> 2009/11/4 Yusuke ENDOH <mame@...>:
>> - Should it be a core-embedded library, not standard library nor gem?
>> - Should it be written in C, not Ruby?
>> - Should it be written from the scratch?  Why don't we import existing
>>  well-tested library?  For example, NArray provides standard deviation.
>
> I believe statistics library is worth to bundled with Ruby.
> But, at first, it should be a gem until it will get stable enough and
> will be widely used.

Such bundling the defact library process may cause
too late to change APIs on bundling it.

I fear that the library is good for usual purpose but has critical
problem for unusual purpose or environments.

> BTW, I think NArray satisfies the conditions; stable enough and widely used.

NArray is stable and widely used.
But I belive its true value is that NArray is the library for libraries.
A Bundled library should be a such library.

--
NARUSE, Yui  <naruse@...>

[Cameron McBride]

On Sat, Nov 7, 2009 at 13:03, NARUSE, Yui <naruse@...> wrote:
>> BTW, I think NArray satisfies the conditions; stable enough and widely used.
>
> NArray is stable and widely used.
> But I belive its true value is that NArray is the library for libraries.
> A Bundled library should be a such library.

I completely agree.

Cameron

[NARUSE, Yui-2]

James Edward Gray II wrote:
> On Nov 3, 2009, at 8:28 PM, Cameron McBride wrote:
>
>> I still don't understand why NArray isn't in the stdlib.  It'd be
>> great to "bless" a simple and elegant library like this to build off
>> of (for specific C extensions, etc).
>
> Yeah, I've always wished it and RBTree could be added to the standard
> library.

RBTree doesn't support Ruby 1.9 .
But it really a library for libraries;
abundled library, SortedSet, uses it.

Moreover to bundle such data structure library encourages Ruby users
to use appropriate data structure.
For example, people who should use RBTree,
but they are just about to play with Hash and Array.
# bundled library by political reason

If someone has responsibility for bundled RBTree in 1.9,
I agree with it.

--
NARUSE, Yui  <naruse@...>