Statistical significance, as Richard wisely reminds us, is not equivalent to
meaningfulness, or substantive significance. Statistical significance just
means that the observed sample difference between something and something
else is not likely to be a fluke while in the whole population the
difference doesn't exist. Given sample size and having chosen a probability
threshold (e.g. p=0.05), statistical tests tell you whether the probability
of getting the observed result by chance is lower or higher than your p. Of
course, a larger sample means you may decide that a smaller difference is
still statistically significant, i.e. that it probably exists also at
population level and not just in your sample. This doesn't make the
difference substantively interesting or meaningful.
Hector
-----Mensaje original-----
De: SPSSX(r) Discussion [mailto:
[hidden email]] En nombre de
Oliver, Richard
Enviado el: Sunday, June 25, 2006 2:31 PM
Para:
[hidden email]
Asunto: Re: Brief Conceptual Question
Okay, I'm not a statistician, so feel free to correct my misconceptions.
In all the undergraduate and graduate statistics courses I took, we were
taught that the results are either significant or not. There was no such
thing as "more" or "less" or "sort of" or "almost" significant. You pick a
significance level and then see what you get. If the p value is at or below
that level, then you reject the null hypothesis.
That's arguably excessively anal and perhaps not very practical in the real
world (but we are talking about an answer to a question in a graduate stats
class), and the same result with samples of differing sizes will yield a
lower p value for larger samples; so in that sense you could say the result
is "more" significant. But perhaps equally important is that larger samples
may yield a "significant" p value (p value less than some arbitrarily
selected value), in instances where smaller samples fail to yield a
significant p value -- and with sufficiently large samples, almost any
difference is statistically significant. But that doesn't necessarily make
it meaningful.
-----Original Message-----
From: SPSSX(r) Discussion [mailto:
[hidden email]] On Behalf Of
Hector Maletta
Sent: Saturday, June 24, 2006 8:15 AM
To:
[hidden email]
Subject: Re: Brief Conceptual Question
Picky indeed, Richard. In my first point, of course I am referring to the
same result, only obtained from two samples of different size. In my second
one, the difference is immaterial for the question asked. In both a priori
and a posteriori interpretations the idea is the same for the purpose of the
question. I tried to keep my answer as simple as possible for the benefit of
our colleague asking the question, who may be confused by too many niceties.
Hector
-----Mensaje original-----
De: Richard Ristow [mailto:
[hidden email]]
Enviado el: Saturday, June 24, 2006 1:48 AM
Para: Hector Maletta;
[hidden email]
Asunto: Re: Brief Conceptual Question
To be a little picky -
At 11:26 PM 6/23/2006, Hector Maletta wrote:
See phrase in brackets and caps:
>1. The larger the sample, the greater the statistical significance of
>a statistical result <OF THE SAME OBSERVED MAGNITUDE, WITH THE SAME
>UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the
>sample, the lower the chance that the result is just a fluke or chance
>occurrence.
>2. If F is above a certain minimum value, you can bet (with a certain
>degree of confidence) that the proportion of variance explained by
>your model is not zero.
Alas, not so; confidence levels tell you something different, and much
less satisfying. What Hector is describing is called the *a posteriori*
probability that you have a false positive result THIS TIME. The
significance level is the *a priori* probability of getting a result
this strong, in the absence of any true underlying effect.
Locked
