Quantcast

Significan ANOVA, but non significant post-hoc

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Significan ANOVA, but non significant post-hoc

ANDRES ALBERTO BURGA LEON

Hello:

I'm running an unbalanced ANOVA, (the variances across groups are homogeneos) and I have a significant F ( p = 0,013) because I'm working with a 5% signifcance level. When I ran the post-hoc test none of the comparations is significant.

I'm not sure how to explain this,

Kindly

Andrés

Mg. Andrés Burga León
Coordinador de Análisis e Informática
Unidad de Medición de la Calidad Educativa
Ministerio de Educación del Perú
Calle El Comercio s/n (espalda del Museo de la Nación)
Lima 41
Perú
Teléfono 615-5840
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

Poling, Taylor Leigh
RE: Significan ANOVA, but non significant post-hoc

If you are running a post-hoc test that adjusts for the # of comparisons you are make (e.g., Bonferroni), and you have several groups, small sample sizes, or both - your post-hoc tests will be a more conservative test of differences between groups. If these are not sig. then I believe it suggests your overall 'significant' F is a type 1 error.

TP


-----Original Message-----
From: SPSSX(r) Discussion on behalf of ANDRES ALBERTO BURGA LEON
Sent: Mon 11/23/2009 10:11 AM
To: [hidden email]
Subject:      Significan ANOVA, but non significant post-hoc

Hello:

I'm running an unbalanced ANOVA, (the variances across groups are
homogeneos) and I have a significant F ( p = 0,013) because I'm working
with a 5% signifcance level. When I ran the post-hoc test none of the
comparations is significant.

I'm not sure how to explain this,

Kindly

Andrés

Mg. Andrés Burga León
Coordinador de Análisis e Informática
Unidad de Medición de la Calidad Educativa
Ministerio de Educación del Perú
Calle El Comercio s/n (espalda del Museo de la Nación)
Lima 41
Perú
Teléfono 615-5840

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

Andrew Lawrence-2
In reply to this post by ANDRES ALBERTO BURGA LEON
Hi Andres,

A little bit more information would be helpful, what is the design of
the ANOVA, is it between subjects (independent samples), within subjects
(repeated measures) or mixed (both)? Are you referring to a main effect
or an interaction? How many groups do you have for the factor(s)? Which
post-hoc tests did you run?

Assuming nothing has gone wrong then this result is not unusual. Please
see this article for a perspective on it:
ftp://ftp.spss.com/pub/spss/statistics/nichols/articles/testsnoa.txt
You can also reference Cardinal & Aitken 2006 ANOVA for the Behavioural
Sciences Researcher p91 (which is where I found the above link).

To paraphrase, non-significant posthoc tests in the presence of a
significant F-test are often interpreted as the effect of a more complex
contrast (such as a linear trend). But really the problem results from a
lack of statistical power to resolve the simple post-hoc comparisons. In
my own experience I find this sort of inconsistency happens with smaller
sample sizes or larger numbers of groups.

all the best,

Andrew



ANDRES ALBERTO BURGA LEON wrote:

>
> Hello:
>
> I'm running an unbalanced ANOVA, (the variances across groups are
> homogeneos) and I have a significant F ( p = 0,013) because I'm
> working with a 5% signifcance level. When I ran the post-hoc test none
> of the comparations is significant.
>
> I'm not sure how to explain this,
>
> Kindly
>
> Andrés
>
> Mg. Andrés Burga León
> Coordinador de Análisis e Informática
> Unidad de Medición de la Calidad Educativa
> Ministerio de Educación del Perú
> Calle El Comercio s/n (espalda del Museo de la Nación)
> Lima 41
> Perú
> Teléfono 615-5840

--
Andrew J. Lawrence
Research Psychologist
Centre for Clinical Neuroscience
St George's University of London
Cranmer Terrace
London
SW17 0RE

[hidden email]
tel: +44(0)20 8266 6468
fax: +44(0)20 8725 2950

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

William B. Ware
In reply to this post by ANDRES ALBERTO BURGA LEON
Andres,

You did not specify what "post-hoc" test you ran... However, if it was a
pair-wise test, it may well be that the contrast being detected by the
anova is a more complex one, such as the difference between (mu1 + mu2)/2
and (m3 + mu4)/2...

wbw

__________________________________________________________________________
William B. Ware, Professor                         Educational Psychology,
CB# 3500                                       Measurement, and Evaluation
University of North Carolina                         PHONE  (919)-962-7848
Chapel Hill, NC      27599-3500                      FAX:   (919)-962-1533
Office:  118 Peabody Hall                            EMAIL: [hidden email]
Adjunct Professor                                    School of Social Work
        Academy of Distinguished Teaching Scholars at UNC-Chapel Hill
__________________________________________________________________________

On Mon, 23 Nov 2009, ANDRES ALBERTO BURGA LEON wrote:

>
> Hello:
>
> I'm running an unbalanced ANOVA, (the variances across groups are
> homogeneos) and I have a significant F ( p = 0,013) because I'm working with
> a 5% signifcance level. When I ran the post-hoc test none of the
> comparations is significant.
>
> I'm not sure how to explain this,
>
> Kindly
>
> Andrés
>
> Mg. Andrés Burga León
> Coordinador de Análisis e Informática
> Unidad de Medición de la Calidad Educativa
> Ministerio de Educación del Perú
> Calle El Comercio s/n (espalda del Museo de la Nación)
> Lima 41
> Perú
> Teléfono 615-5840
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

Arthur Burke-2
In reply to this post by ANDRES ALBERTO BURGA LEON
Andres ... The overall ANOVA F test identifies significant differences between means.  However, the F test will identify significant differences for combinations of means as well as differences between pairs of means.  That is why an overall F test can be significant, yet none of the pairwise differences are significant.
 
Art

*************************

Art Burke

Associate, Evaluation Program

Education Northwest

101 SW Main St, Ste 500

Portland, OR 97204

Phone: 503.275.9592

Art.Burke[hidden email]

http://educationnorthwest.org

We have recently changed our name to "Education Northwest" from "Northwest Regional Educational Laboratory." Please note the new e-mail and Web addresses in the signature above. You may continue to find us on the Web at http://www.nwrel.org for the immediate future as well.

************************



From: ANDRES ALBERTO BURGA LEON [mailto:[hidden email]]
Sent: Monday, November 23, 2009 7:12 AM
To: [hidden email]
Subject: Significan ANOVA, but non significant post-hoc


Hello:

I'm running an unbalanced ANOVA, (the variances across groups are homogeneos) and I have a significant F ( p = 0,013) because I'm working with a 5% signifcance level. When I ran the post-hoc test none of the comparations is significant.

I'm not sure how to explain this,

Kindly

Andrés

Mg. Andrés Burga León
Coordinador de Análisis e Informática
Unidad de Medición de la Calidad Educativa
Ministerio de Educación del Perú
Calle El Comercio s/n (espalda del Museo de la Nación)
Lima 41
Perú
Teléfono 615-5840
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

Steve Simon, P.Mean Consulting
In reply to this post by ANDRES ALBERTO BURGA LEON
ANDRES ALBERTO BURGA LEON wrote:

> I'm running an unbalanced ANOVA, (the variances across groups are
> homogeneos) and I have a significant F ( p = 0,013) because I'm working
> with a 5% signifcance level. When I ran the post-hoc test none of the
> comparations is significant.
>
> I'm not sure how to explain this,

In addition to the comments made so far, note
  * http://www.pmean.com/05/TukeyTest.html
which has a link to a previous discussion of this issue in STAT-L.

But the unbalanced nature of the data can create extra problems, beyond
what I commented on in my webpage. Many post hoc tests were designed for
balanced data. A common fix is to replace the individual sample sizes
with the harmonic mean of the sample sizes, but this will produce, at
best, an approximate solution.
--
Steve Simon, Standard Disclaimer
The Monthly Mean is celebrating its first anniversary.
Find out more about the newsletter that dares
to call itself "average" at www.pmean.com/news

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Significan ANOVA, but non significant post-hoc

ANDRES ALBERTO BURGA LEON
In reply to this post by William B. Ware

Thanks everybody for your comments.

I’m giving more information about the problem I’m trying to solve:

I have a random sample of 677 mothers who answer open-ended question: “Do you think initial education is important? Why?”

Those answers were subject to content analysis and we have 8 different groups, whose n are as follows:

General
220
Confianza
117
Afectos
104
Aprendizaje
48
Confianza y afectos
32
Diversión
35
Otros
90
No es importante
31

I also have a student achievement measure. I want to investigate if there are significant differences on students mean achievement, considering their mothers conceptions about the importance of initial education.

So I run an ANOVA and I check that the variances are homogeneous across groups. After that I use the Hochberg GT2 post hoc test. I’ve read that this is a recommended post-hoc test when the sample sizes are unequal.

Andrés


Mg. Andrés Burga León
Coordinador de Análisis e Informática
Unidad de Medición de la Calidad Educativa
Ministerio de Educación del Perú
Calle El Comercio s/n (espalda del Museo de la Nación)
Lima 41
Perú
Teléfono 615-5840

Loading...