is AIC always 100% in evaluating a model?

[alexander russell-2]

Hello,
I'd like to say that it's clear when an independent variable can be ruled
out generally speaking; on the other hand in R's AIC with bbmle, if one
finds a better AIC value for a model without the given independent variable,
versus the same model with, can we say that the independent variable is not
likely to be significant(in the ordinary sense!)?

That is, having made a lot of models from a data set, then the best two are
say 78.2 and 79.3 without and with (a second independent variable
respectively) should we say it's better to judge the influence of the 2nd IV
as insignificant?
regards,
-shfets

        [[alternative HTML version deleted]]

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Ben Bolker]

alexander russell-2 wrote:

Hello,
I'd like to say that it's clear when an independent variable can be ruled
out generally speaking; on the other hand in R's AIC with bbmle, if one
finds a better AIC value for a model without the given independent variable,
versus the same model with, can we say that the independent variable is not
likely to be significant(in the ordinary sense!)?

That is, having made a lot of models from a data set, then the best two are
say 78.2 and 79.3 without and with (a second independent variable
respectively) should we say it's better to judge the influence of the 2nd IV
as insignificant?
regards,
-shfets
_____________________________________

Without meaning to sound snarky, it's best not to consider hypothesis
testing (statistical significance) and AIC in the same analysis.
If you want to decide whether predictor variables have a significant
effect on a response, you should consider their effect in the full model,
via Wald test, likelihood ratio test, etc..  If you want to find the model
with the best expected predictive capability (i.e. lowest expected
Kullback-Leibler distance), you should use AIC.

  Burnham and Anderson, among others, say this repeatedly.

  In general, for a one-parameter difference, hypothesis testing
is "more conservative" than AIC (e.g., critical log-likelihood difference
for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
difference required to say that a model is expected to have better
predictive capability/lower AIC is 1) -- but since they are designed to answer
such different questions, it's not even a fair comparison.

  Ben Bolker

[Tal Galili]

Hi Ben,
I just wished to give a small remark about your claim:
"it's best not to consider hypothesis testing (statistical significance) and
AIC in the same analysis."

Since in the case of forward selection for orthogonal matrix's, it can be
shown that AIC is like using a P to enter rule of 0.16.  For further
reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED
ONFALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1239888367


Cheers,
Tal Galili





On Sat, Jul 4, 2009 at 1:46 AM, Ben Bolker <bolker@...> wrote:

>
>
>
> alexander russell-2 wrote:
> >
> > Hello,
> > I'd like to say that it's clear when an independent variable can be ruled
> > out generally speaking; on the other hand in R's AIC with bbmle, if one
> > finds a better AIC value for a model without the given independent
> > variable,
> > versus the same model with, can we say that the independent variable is
> > not
> > likely to be significant(in the ordinary sense!)?
> >
> > That is, having made a lot of models from a data set, then the best two
> > are
> > say 78.2 and 79.3 without and with (a second independent variable
> > respectively) should we say it's better to judge the influence of the 2nd
> > IV
> > as insignificant?
> > regards,
> > -shfets
> > _____________________________________
> >
> >
>
> Without meaning to sound snarky, it's best not to consider hypothesis
> testing (statistical significance) and AIC in the same analysis.
> If you want to decide whether predictor variables have a significant
> effect on a response, you should consider their effect in the full model,
> via Wald test, likelihood ratio test, etc..  If you want to find the model
> with the best expected predictive capability (i.e. lowest expected
> Kullback-Leibler distance), you should use AIC.
>
>  Burnham and Anderson, among others, say this repeatedly.
>
>  In general, for a one-parameter difference, hypothesis testing
> is "more conservative" than AIC (e.g., critical log-likelihood difference
> for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
> difference required to say that a model is expected to have better
> predictive capability/lower AIC is 1) -- but since they are designed to
> answer
> such different questions, it's not even a fair comparison.
>
>  Ben Bolker
>
> --
> View this message in context:
> http://www.nabble.com/is-AIC-always-100--in-evaluating-a-model--tp24323538p24329622.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help@... mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

--
----------------------------------------------


My contact information:
Tal Galili
Phone number: 972-50-3373767
FaceBook: Tal Galili
My Blogs:
http://www.r-statistics.com/
http://www.talgalili.com
http://www.biostatistics.co.il

        [[alternative HTML version deleted]]

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Ben Bolker]

Tal Galili wrote:
> Hi Ben,
> I just wished to give a small remark about your claim:
> "it's best not to consider hypothesis testing (statistical significance) and AIC in the same analysis."
>
> Since in the case of forward selection for orthogonal matrix's, it can be shown that AIC is like using a P to enter rule of 0.16.  For further reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED ON
> FALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
> http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1239888367
>
>

  Haven't read the paper yet, but I would say that makes sense --

> pchisq(3.84,1,lower.tail=FALSE)
[1] 0.05004352
> pchisq(2,1,lower.tail=FALSE)
[1] 0.1572992


--
Ben Bolker
Associate professor, Biology Dep't, Univ. of Florida
bolker@... / www.zoology.ufl.edu/bolker
GPG key: www.zoology.ufl.edu/bolker/benbolker-publickey.asc



______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Frank E Harrell Jr]

Tal Galili wrote:

> Hi Ben,
> I just wished to give a small remark about your claim:
> "it's best not to consider hypothesis testing (statistical significance) and
> AIC in the same analysis."
>
> Since in the case of forward selection for orthogonal matrix's, it can be
> shown that AIC is like using a P to enter rule of 0.16.  For further
> reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED
> ONFALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
> http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoas/1239888367
>
>
> Cheers,
> Tal Galili

Tal,

That is not limited to orthogonal designs.  When used for one variable
at a time variable selection. AIC is just a restatement of the P-value,
and as such, doesn't solve the severe problems with stepwise variable
selection other than forcing us to use slightly more sensible alpha
values.  As an aside, some statisticians try to deal with multiplicity
problems caused by stepwise variable selection by making alpha smaller
than 0.05.  This increases bias by giving variables whose effects are
estimated with error a greater relative chance of being selected.  alpha
typically needs to be 0.5 or greater to avoid problems with stepwise
variable selection.

AIC was designed to compare two pre-specified models.

Variable selection does not compete well with shrinkage methods that
simultaneously model all potential predictors.

Frank

>
>
>
>
>
> On Sat, Jul 4, 2009 at 1:46 AM, Ben Bolker <bolker@...> wrote:
>
>>
>>
>> alexander russell-2 wrote:
>>> Hello,
>>> I'd like to say that it's clear when an independent variable can be ruled
>>> out generally speaking; on the other hand in R's AIC with bbmle, if one
>>> finds a better AIC value for a model without the given independent
>>> variable,
>>> versus the same model with, can we say that the independent variable is
>>> not
>>> likely to be significant(in the ordinary sense!)?
>>>
>>> That is, having made a lot of models from a data set, then the best two
>>> are
>>> say 78.2 and 79.3 without and with (a second independent variable
>>> respectively) should we say it's better to judge the influence of the 2nd
>>> IV
>>> as insignificant?
>>> regards,
>>> -shfets
>>> _____________________________________
>>>
>>>
>> Without meaning to sound snarky, it's best not to consider hypothesis
>> testing (statistical significance) and AIC in the same analysis.
>> If you want to decide whether predictor variables have a significant
>> effect on a response, you should consider their effect in the full model,
>> via Wald test, likelihood ratio test, etc..  If you want to find the model
>> with the best expected predictive capability (i.e. lowest expected
>> Kullback-Leibler distance), you should use AIC.
>>
>>  Burnham and Anderson, among others, say this repeatedly.
>>
>>  In general, for a one-parameter difference, hypothesis testing
>> is "more conservative" than AIC (e.g., critical log-likelihood difference
>> for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
>> difference required to say that a model is expected to have better
>> predictive capability/lower AIC is 1) -- but since they are designed to
>> answer
>> such different questions, it's not even a fair comparison.
>>
>>  Ben Bolker
>>
>> --
>> View this message in context:
>> http://www.nabble.com/is-AIC-always-100--in-evaluating-a-model--tp24323538p24329622.html
>> Sent from the R help mailing list archive at Nabble.com.
>>
>> ______________________________________________
>> R-help@... mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
>
>

--
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Ravi Varadhan]

But you are still left with the problem of choosing the regularization parameter, i.e. how much to shrink the coefficients?  In other words, there is no free ride.

Ravi.

____________________________________________________________________

Ravi Varadhan, Ph.D.
Assistant Professor,
Division of Geriatric Medicine and Gerontology
School of Medicine
Johns Hopkins University

Ph. [phone]
email: rvaradhan@...


----- Original Message -----
From: Frank E Harrell Jr <f.harrell@...>
Date: Saturday, July 4, 2009 9:26 am
Subject: Re: [R] is AIC always 100% in evaluating a model?
To: Tal Galili <tal.galili@...>
Cc: r-help@..., Ben Bolker <bolker@...>

> Tal Galili wrote:
>  > Hi Ben,
>  > I just wished to give a small remark about your claim:
>  > "it's best not to consider hypothesis testing (statistical
> significance) and
>  > AIC in the same analysis."
>  >
>  > Since in the case of forward selection for orthogonal matrix's, it
> can be
>  > shown that AIC is like using a P to enter rule of 0.16.  For further
>  > reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED
>  > ONFALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
>  >
>  >
>  >
>  > Cheers,
>  > Tal Galili
>  
>  Tal,
>  
>  That is not limited to orthogonal designs.  When used for one
> variable
>  at a time variable selection. AIC is just a restatement of the
> P-value,
>  and as such, doesn't solve the severe problems with stepwise variable
>
>  selection other than forcing us to use slightly more sensible alpha
>  values.  As an aside, some statisticians try to deal with
> multiplicity
>  problems caused by stepwise variable selection by making alpha
> smaller
>  than 0.05.  This increases bias by giving variables whose effects are
>
>  estimated with error a greater relative chance of being selected.  
> alpha
>  typically needs to be 0.5 or greater to avoid problems with stepwise
>
>  variable selection.
>  
>  AIC was designed to compare two pre-specified models.
>  
>  Variable selection does not compete well with shrinkage methods that
>
>  simultaneously model all potential predictors.
>  
>  Frank
>  
>  >
>  >
>  >
>  >
>  >
>  > On Sat, Jul 4, 2009 at 1:46 AM, Ben Bolker <bolker@...> wrote:
>  >
>  >>
>  >>
>  >> alexander russell-2 wrote:
>  >>> Hello,
>  >>> I'd like to say that it's clear when an independent variable can
> be ruled
>  >>> out generally speaking; on the other hand in R's AIC with bbmle,
> if one
>  >>> finds a better AIC value for a model without the given independent
>  >>> variable,
>  >>> versus the same model with, can we say that the independent
> variable is
>  >>> not
>  >>> likely to be significant(in the ordinary sense!)?
>  >>>
>  >>> That is, having made a lot of models from a data set, then the
> best two
>  >>> are
>  >>> say 78.2 and 79.3 without and with (a second independent variable
>  >>> respectively) should we say it's better to judge the influence of
> the 2nd
>  >>> IV
>  >>> as insignificant?
>  >>> regards,
>  >>> -shfets
>  >>> _____________________________________
>  >>>
>  >>>
>  >> Without meaning to sound snarky, it's best not to consider hypothesis
>  >> testing (statistical significance) and AIC in the same analysis.
>  >> If you want to decide whether predictor variables have a significant
>  >> effect on a response, you should consider their effect in the full
> model,
>  >> via Wald test, likelihood ratio test, etc..  If you want to find
> the model
>  >> with the best expected predictive capability (i.e. lowest expected
>  >> Kullback-Leibler distance), you should use AIC.
>  >>
>  >>  Burnham and Anderson, among others, say this repeatedly.
>  >>
>  >>  In general, for a one-parameter difference, hypothesis testing
>  >> is "more conservative" than AIC (e.g., critical log-likelihood difference
>  >> for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
>  >> difference required to say that a model is expected to have better
>  >> predictive capability/lower AIC is 1) -- but since they are
> designed to
>  >> answer
>  >> such different questions, it's not even a fair comparison.
>  >>
>  >>  Ben Bolker
>  >>
>  >> --
>  >> View this message in context:
>  >>
>  >> Sent from the R help mailing list archive at Nabble.com.
>  >>
>  >> ______________________________________________
>  >> R-help@... mailing list
>  >>
>  >> PLEASE do read the posting guide
>  >>
>  >> and provide commented, minimal, self-contained, reproducible code.
>  >>
>  >
>  >
>  >
>  
>  
>  --
>  Frank E Harrell Jr   Professor and Chair           School of Medicine
>                        Department of Biostatistics   Vanderbilt University
>  
>  ______________________________________________
>  R-help@... mailing list
>  
>  PLEASE do read the posting guide
>  and provide commented, minimal, self-contained, reproducible code.

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Frank E Harrell Jr]

Ravi Varadhan wrote:
> But you are still left with the problem of choosing the regularization parameter, i.e. how much to shrink the coefficients?  In other words, there is no free ride.
>
> Ravi.

Ravi,

Choosing a penalty factor is extremely easy compared to variable
selection.  There is a unique optimum and, depending on your model, only
a single number to solve for.  With stepwise variable selection
variables move in and out of the model in an extremely non-monotonic way.

One good way to choose the penalty is to use the effective AIC.  An
example is at http://biostat.mc.vanderbilt.edu/rms .  AIC is good to use
in that context where the solution path is simple, unlike stepwise
selection.

In the end, shrinkage beats standard variable selection easily with
regard to predictive accuracy, discrimination, and adequate adjustment
for confounding.

Frank

>
> ____________________________________________________________________
>
> Ravi Varadhan, Ph.D.
> Assistant Professor,
> Division of Geriatric Medicine and Gerontology
> School of Medicine
> Johns Hopkins University
>
> Ph. [phone]
> email: rvaradhan@...
>
>
> ----- Original Message -----
> From: Frank E Harrell Jr <f.harrell@...>
> Date: Saturday, July 4, 2009 9:26 am
> Subject: Re: [R] is AIC always 100% in evaluating a model?
> To: Tal Galili <tal.galili@...>
> Cc: r-help@..., Ben Bolker <bolker@...>
>
>
>> Tal Galili wrote:
>>  > Hi Ben,
>>  > I just wished to give a small remark about your claim:
>>  > "it's best not to consider hypothesis testing (statistical
>> significance) and
>>  > AIC in the same analysis."
>>  >
>>  > Since in the case of forward selection for orthogonal matrix's, it
>> can be
>>  > shown that AIC is like using a P to enter rule of 0.16.  For further
>>  > reference see:page 3 of: "A SIMPLE FORWARD SELECTION PROCEDURE BASED
>>  > ONFALSE DISCOVERY RATE CONTROL" BY YOAV BENJAMINI AND YULIA GAVRILOV,
>>  >
>>  >
>>  >
>>  > Cheers,
>>  > Tal Galili
>>  
>>  Tal,
>>  
>>  That is not limited to orthogonal designs.  When used for one
>> variable
>>  at a time variable selection. AIC is just a restatement of the
>> P-value,
>>  and as such, doesn't solve the severe problems with stepwise variable
>>
>>  selection other than forcing us to use slightly more sensible alpha
>>  values.  As an aside, some statisticians try to deal with
>> multiplicity
>>  problems caused by stepwise variable selection by making alpha
>> smaller
>>  than 0.05.  This increases bias by giving variables whose effects are
>>
>>  estimated with error a greater relative chance of being selected.  
>> alpha
>>  typically needs to be 0.5 or greater to avoid problems with stepwise
>>
>>  variable selection.
>>  
>>  AIC was designed to compare two pre-specified models.
>>  
>>  Variable selection does not compete well with shrinkage methods that
>>
>>  simultaneously model all potential predictors.
>>  
>>  Frank
>>  
>>  >
>>  >
>>  >
>>  >
>>  >
>>  > On Sat, Jul 4, 2009 at 1:46 AM, Ben Bolker <bolker@...> wrote:
>>  >
>>  >>
>>  >>
>>  >> alexander russell-2 wrote:
>>  >>> Hello,
>>  >>> I'd like to say that it's clear when an independent variable can
>> be ruled
>>  >>> out generally speaking; on the other hand in R's AIC with bbmle,
>> if one
>>  >>> finds a better AIC value for a model without the given independent
>>  >>> variable,
>>  >>> versus the same model with, can we say that the independent
>> variable is
>>  >>> not
>>  >>> likely to be significant(in the ordinary sense!)?
>>  >>>
>>  >>> That is, having made a lot of models from a data set, then the
>> best two
>>  >>> are
>>  >>> say 78.2 and 79.3 without and with (a second independent variable
>>  >>> respectively) should we say it's better to judge the influence of
>> the 2nd
>>  >>> IV
>>  >>> as insignificant?
>>  >>> regards,
>>  >>> -shfets
>>  >>> _____________________________________
>>  >>>
>>  >>>
>>  >> Without meaning to sound snarky, it's best not to consider hypothesis
>>  >> testing (statistical significance) and AIC in the same analysis.
>>  >> If you want to decide whether predictor variables have a significant
>>  >> effect on a response, you should consider their effect in the full
>> model,
>>  >> via Wald test, likelihood ratio test, etc..  If you want to find
>> the model
>>  >> with the best expected predictive capability (i.e. lowest expected
>>  >> Kullback-Leibler distance), you should use AIC.
>>  >>
>>  >>  Burnham and Anderson, among others, say this repeatedly.
>>  >>
>>  >>  In general, for a one-parameter difference, hypothesis testing
>>  >> is "more conservative" than AIC (e.g., critical log-likelihood difference
>>  >> for a p-value of 0.05 under the LRT test is 1.92, while the log-likelihood
>>  >> difference required to say that a model is expected to have better
>>  >> predictive capability/lower AIC is 1) -- but since they are
>> designed to
>>  >> answer
>>  >> such different questions, it's not even a fair comparison.
>>  >>
>>  >>  Ben Bolker
>>  >>
>>  >> --
>>  >> View this message in context:
>>  >>
>>  >> Sent from the R help mailing list archive at Nabble.com.
>>  >>
>>  >> ______________________________________________
>>  >> R-help@... mailing list
>>  >>
>>  >> PLEASE do read the posting guide
>>  >>
>>  >> and provide commented, minimal, self-contained, reproducible code.
>>  >>
>>  >
>>  >
>>  >
>>  
>>  
>>  --
>>  Frank E Harrell Jr   Professor and Chair           School of Medicine
>>                        Department of Biostatistics   Vanderbilt University
>>  
>>  ______________________________________________
>>  R-help@... mailing list
>>  
>>  PLEASE do read the posting guide
>>  and provide commented, minimal, self-contained, reproducible code.
>

--
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Tal Galili]

Hello Frank,

Thank you for the extension and remarks.
The basic weakness of stepwise regression VS going through all-subsets is
very much agreed upon.  Although from what I gather there is one case where
all subsets will be a problem to implement, that is for very LARGE datasets
- especially in the sense of a lot of explanatory variables, and also with
regards to cases where we have more explanatory variables then data points.
In such cases I wonder if using stepwise regression could be found to be
more realistic to implement then all subsets checks.
Then again, I imagine (although not from real experience) that shrinkage
methods (used with LARS) could be practical in those cases too.



I am looking forward to meeting you on Tuesday and taking your first
tutorial of the day,

With regard,
Tal







On Sat, Jul 4, 2009 at 4:22 PM, Frank E Harrell Jr <f.harrell@...
> wrote:

> sed for one variable at a time variable selection. AIC is just a
> restatement of the P-value, and as such, doesn't solve the severe problems
> with stepwise v




--
----------------------------------------------


My contact information:
Tal Galili
Phone number: 972-50-3373767
FaceBook: Tal Galili
My Blogs:
http://www.r-statistics.com/
http://www.talgalili.com
http://www.biostatistics.co.il

        [[alternative HTML version deleted]]

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

[Frank E Harrell Jr]

Tal Galili wrote:

> Hello Frank,
>
> Thank you for the extension and remarks.
> The basic weakness of stepwise regression VS going through all-subsets
> is very much agreed upon.  Although from what I gather there is one case
> where all subsets will be a problem to implement, that is for very LARGE
> datasets - especially in the sense of a lot of explanatory variables,
> and also with regards to cases where we have more explanatory variables
> then data points.
> In such cases I wonder if using stepwise regression could be found to be
> more realistic to implement then all subsets checks.
> Then again, I imagine (although not from real experience) that shrinkage
> methods (used with LARS) could be practical in those cases too.
>
>
>
> I am looking forward to meeting you on Tuesday and taking your first
> tutorial of the day,

I look forward to seeing you in Rennes.

>
> With regard,
> Tal

All subsets regression is an especially bad form of stepwise regression.
  It has terrible operating characteristics.

Cheers,
Frank

>
>
>
>
>
>
>
> On Sat, Jul 4, 2009 at 4:22 PM, Frank E Harrell Jr
> <f.harrell@... <mailto:f.harrell@...>> wrote:
>
>     sed for one variable at a time variable selection. AIC is just a
>     restatement of the P-value, and as such, doesn't solve the severe
>     problems with stepwise v
>
>
>
>
> --
> ----------------------------------------------
>
>
> My contact information:
> Tal Galili
> Phone number: 972-50-3373767
> FaceBook: Tal Galili
> My Blogs:
> http://www.r-statistics.com/
> http://www.talgalili.com
> http://www.biostatistics.co.il
>
>

--
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

______________________________________________
R-help@... mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.