Brief Conceptual Question

7 messages
Open this post in threaded view
|
Report Content as Inappropriate

Brief Conceptual Question

 Dear Colleagues,                          I am a Master's candidate in Psychology, and I'm currently studying for an exam about statistics and research methodology. There was just one conceptual question that I could not seem to find an adequate answer to. The question is as follows: Explain the relationship between sample size and statistical significance testing. Why is the size of an F ratio is bigger with a larger sample size? What does this imply?                      I very much appreciate any feedback anyone can offer! Thank you in advance for your responses. Have a wonderful weekend. Best, Carrie Margulies
Open this post in threaded view
|
Report Content as Inappropriate

Re: Brief Conceptual Question

 Carrie, First of all I'd advice you to read some book on statistics and research methodology. It is the best course, all told, for an exam on statistics and research methodology. However, that said, this may be useful: 1. The larger the sample, the greater the statistical significance of a statistical result. This means that the larger the sample, the lower the chance that the result is just a fluke or chance occurrence. 2. In fact an F ratio is NOT larger for larger sample sizes. The F ratio is the ratio of explained variance to residual variance, and it does not depend on sample size. It depends on the proportion of variance in the dependent variable that is explained by independent or predictor variables. If F is above a certain minimum value, you can bet (with a certain degree of confidence) that the proportion of variance explained by your model is not zero. In any case, for any given degree of confidence, this minimum F ratio you need for a result to be significant is LOWER with larger samples. So either the second part of your exam question is a malicious trick by your teacher, or you have transcribed it wrong. Hector -----Mensaje original----- De: SPSSX(r) Discussion [mailto:[hidden email]] En nombre de Carrie Margulies Enviado el: Friday, June 23, 2006 11:43 PM Para: [hidden email] Asunto: Brief Conceptual Question Dear Colleagues,                          I am a Master's candidate in Psychology, and I'm currently studying for an exam about statistics and research methodology. There was just one conceptual question that I could not seem to find an adequate answer to. The question is as follows: Explain the relationship between sample size and statistical significance testing. Why is the size of an F ratio is bigger with a larger sample size? What does this imply?                      I very much appreciate any feedback anyone can offer! Thank you in advance for your responses. Have a wonderful weekend. Best, Carrie Margulies
Open this post in threaded view
|
Report Content as Inappropriate

Re: Brief Conceptual Question

 To be a little picky - At 11:26 PM 6/23/2006, Hector Maletta wrote: See phrase in brackets and caps: >1. The larger the sample, the greater the statistical significance of >a statistical result UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the >sample, the lower the chance that the result is just a fluke or chance >occurrence. >2. If F is above a certain minimum value, you can bet (with a certain >degree of confidence) that the proportion of variance explained by >your model is not zero. Alas, not so; confidence levels tell you something different, and much less satisfying. What Hector is describing is called the *a posteriori* probability that you have a false positive result THIS TIME. The significance level is the *a priori* probability of getting a result this strong, in the absence of any true underlying effect.
Open this post in threaded view
|
Report Content as Inappropriate

Re: Brief Conceptual Question

 Picky indeed, Richard. In my first point, of course I am referring to the same result, only obtained from two samples of different size. In my second one, the difference is immaterial for the question asked. In both a priori and a posteriori interpretations the idea is the same for the purpose of the question. I tried to keep my answer as simple as possible for the benefit of our colleague asking the question, who may be confused by too many niceties. Hector -----Mensaje original----- De: Richard Ristow [mailto:[hidden email]] Enviado el: Saturday, June 24, 2006 1:48 AM Para: Hector Maletta; [hidden email] Asunto: Re: Brief Conceptual Question To be a little picky - At 11:26 PM 6/23/2006, Hector Maletta wrote: See phrase in brackets and caps: >1. The larger the sample, the greater the statistical significance of >a statistical result UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the >sample, the lower the chance that the result is just a fluke or chance >occurrence. >2. If F is above a certain minimum value, you can bet (with a certain >degree of confidence) that the proportion of variance explained by >your model is not zero. Alas, not so; confidence levels tell you something different, and much less satisfying. What Hector is describing is called the *a posteriori* probability that you have a false positive result THIS TIME. The significance level is the *a priori* probability of getting a result this strong, in the absence of any true underlying effect.
Open this post in threaded view
|
Report Content as Inappropriate

Re: Brief Conceptual Question

 In reply to this post by Carrie Margulies Okay, I'm not a statistician, so feel free to correct my misconceptions. In all the undergraduate and graduate statistics courses I took, we were taught that the results are either significant or not. There was no such thing as "more" or "less" or "sort of" or "almost" significant. You pick a significance level and then see what you get. If the p value is at or below that level, then you reject the null hypothesis. That's arguably excessively anal and perhaps not very practical in the real world (but we are talking about an answer to a question in a graduate stats class), and the same result with samples of differing sizes will yield a lower p value for larger samples; so in that sense you could say the result is "more" significant. But perhaps equally important is that larger samples may yield a "significant" p value (p value less than some arbitrarily selected value), in instances where smaller samples fail to yield a significant p value -- and with sufficiently large samples, almost any difference is statistically significant. But that doesn't necessarily make it meaningful. -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta Sent: Saturday, June 24, 2006 8:15 AM To: [hidden email] Subject: Re: Brief Conceptual Question Picky indeed, Richard. In my first point, of course I am referring to the same result, only obtained from two samples of different size. In my second one, the difference is immaterial for the question asked. In both a priori and a posteriori interpretations the idea is the same for the purpose of the question. I tried to keep my answer as simple as possible for the benefit of our colleague asking the question, who may be confused by too many niceties. Hector -----Mensaje original----- De: Richard Ristow [mailto:[hidden email]] Enviado el: Saturday, June 24, 2006 1:48 AM Para: Hector Maletta; [hidden email] Asunto: Re: Brief Conceptual Question To be a little picky - At 11:26 PM 6/23/2006, Hector Maletta wrote: See phrase in brackets and caps: >1. The larger the sample, the greater the statistical significance of >a statistical result UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the >sample, the lower the chance that the result is just a fluke or chance >occurrence. >2. If F is above a certain minimum value, you can bet (with a certain >degree of confidence) that the proportion of variance explained by >your model is not zero. Alas, not so; confidence levels tell you something different, and much less satisfying. What Hector is describing is called the *a posteriori* probability that you have a false positive result THIS TIME. The significance level is the *a priori* probability of getting a result this strong, in the absence of any true underlying effect.
Open this post in threaded view
|
Report Content as Inappropriate

Re: Brief Conceptual Question

 Statistical significance, as Richard wisely reminds us, is not equivalent to meaningfulness, or substantive significance. Statistical significance just means that the observed sample difference between something and something else is not likely to be a fluke while in the whole population the difference doesn't exist. Given sample size and having chosen a probability threshold (e.g. p=0.05), statistical tests tell you whether the probability of getting the observed result by chance is lower or higher than your p. Of course, a larger sample means you may decide that a smaller difference is still statistically significant, i.e. that it probably exists also at population level and not just in your sample. This doesn't make the difference substantively interesting or meaningful. Hector -----Mensaje original----- De: SPSSX(r) Discussion [mailto:[hidden email]] En nombre de Oliver, Richard Enviado el: Sunday, June 25, 2006 2:31 PM Para: [hidden email] Asunto: Re: Brief Conceptual Question Okay, I'm not a statistician, so feel free to correct my misconceptions. In all the undergraduate and graduate statistics courses I took, we were taught that the results are either significant or not. There was no such thing as "more" or "less" or "sort of" or "almost" significant. You pick a significance level and then see what you get. If the p value is at or below that level, then you reject the null hypothesis. That's arguably excessively anal and perhaps not very practical in the real world (but we are talking about an answer to a question in a graduate stats class), and the same result with samples of differing sizes will yield a lower p value for larger samples; so in that sense you could say the result is "more" significant. But perhaps equally important is that larger samples may yield a "significant" p value (p value less than some arbitrarily selected value), in instances where smaller samples fail to yield a significant p value -- and with sufficiently large samples, almost any difference is statistically significant. But that doesn't necessarily make it meaningful. -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Hector Maletta Sent: Saturday, June 24, 2006 8:15 AM To: [hidden email] Subject: Re: Brief Conceptual Question Picky indeed, Richard. In my first point, of course I am referring to the same result, only obtained from two samples of different size. In my second one, the difference is immaterial for the question asked. In both a priori and a posteriori interpretations the idea is the same for the purpose of the question. I tried to keep my answer as simple as possible for the benefit of our colleague asking the question, who may be confused by too many niceties. Hector -----Mensaje original----- De: Richard Ristow [mailto:[hidden email]] Enviado el: Saturday, June 24, 2006 1:48 AM Para: Hector Maletta; [hidden email] Asunto: Re: Brief Conceptual Question To be a little picky - At 11:26 PM 6/23/2006, Hector Maletta wrote: See phrase in brackets and caps: >1. The larger the sample, the greater the statistical significance of >a statistical result UNEXPLAINED VARIANCE IN THE DATA>. This means that the larger the >sample, the lower the chance that the result is just a fluke or chance >occurrence. >2. If F is above a certain minimum value, you can bet (with a certain >degree of confidence) that the proportion of variance explained by >your model is not zero. Alas, not so; confidence levels tell you something different, and much less satisfying. What Hector is describing is called the *a posteriori* probability that you have a false positive result THIS TIME. The significance level is the *a priori* probability of getting a result this strong, in the absence of any true underlying effect.