Statistical significance, as Richard wisely reminds us, is not equivalent to

meaningfulness, or substantive significance. Statistical significance just

means that the observed sample difference between something and something

else is not likely to be a fluke while in the whole population the

difference doesn't exist. Given sample size and having chosen a probability

threshold (e.g. p=0.05), statistical tests tell you whether the probability

of getting the observed result by chance is lower or higher than your p. Of

course, a larger sample means you may decide that a smaller difference is

still statistically significant, i.e. that it probably exists also at

population level and not just in your sample. This doesn't make the

difference substantively interesting or meaningful.

Hector

-----Mensaje original-----

De: SPSSX(r) Discussion [mailto:

[hidden email]] En nombre de

Oliver, Richard

Enviado el: Sunday, June 25, 2006 2:31 PM

Para:

[hidden email]
Asunto: Re: Brief Conceptual Question

Okay, I'm not a statistician, so feel free to correct my misconceptions.

In all the undergraduate and graduate statistics courses I took, we were

taught that the results are either significant or not. There was no such

thing as "more" or "less" or "sort of" or "almost" significant. You pick a

significance level and then see what you get. If the p value is at or below

that level, then you reject the null hypothesis.

That's arguably excessively anal and perhaps not very practical in the real

world (but we are talking about an answer to a question in a graduate stats

class), and the same result with samples of differing sizes will yield a

lower p value for larger samples; so in that sense you could say the result

is "more" significant. But perhaps equally important is that larger samples

may yield a "significant" p value (p value less than some arbitrarily

selected value), in instances where smaller samples fail to yield a

significant p value -- and with sufficiently large samples, almost any

difference is statistically significant. But that doesn't necessarily make

it meaningful.

-----Original Message-----

From: SPSSX(r) Discussion [mailto:

[hidden email]] On Behalf Of

Hector Maletta

Sent: Saturday, June 24, 2006 8:15 AM

To:

[hidden email]
Subject: Re: Brief Conceptual Question

Picky indeed, Richard. In my first point, of course I am referring to the

same result, only obtained from two samples of different size. In my second

one, the difference is immaterial for the question asked. In both a priori

and a posteriori interpretations the idea is the same for the purpose of the

question. I tried to keep my answer as simple as possible for the benefit of

our colleague asking the question, who may be confused by too many niceties.

Hector

-----Mensaje original-----

De: Richard Ristow [mailto:

[hidden email]]

Enviado el: Saturday, June 24, 2006 1:48 AM

Para: Hector Maletta;

[hidden email]
Asunto: Re: Brief Conceptual Question

To be a little picky -

At 11:26 PM 6/23/2006, Hector Maletta wrote:

See phrase in brackets and caps:

>1. The larger the sample, the greater the statistical significance of

>a statistical result <OF THE SAME OBSERVED MAGNITUDE, WITH THE SAME

>UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the

>sample, the lower the chance that the result is just a fluke or chance

>occurrence.

>2. If F is above a certain minimum value, you can bet (with a certain

>degree of confidence) that the proportion of variance explained by

>your model is not zero.

Alas, not so; confidence levels tell you something different, and much

less satisfying. What Hector is describing is called the *a posteriori*

probability that you have a false positive result THIS TIME. The

significance level is the *a priori* probability of getting a result

this strong, in the absence of any true underlying effect.