significant F change, but nonsignificant regression model overall

24 messages
12
Open this post in threaded view
|
Report Content as Inappropriate

significant F change, but nonsignificant regression model overall

 Hi everyone, One of our post-docs is having troubles with a regression model he is trying to run.  He is trying to predict outcome in babies based on some pregnancy variables from the mothers collected during gestation.  He has 75 participants.  He has entered four variables to control for on the first step, and then two other predictors on the 2nd step.  So we're trying to see if these two predictors are significant above and beyond the four variables we are controlling for on the first step of the regression.   The F change for adding these 2 predictors on Step 2 is significant - and the R2 change is .19 so not big, but not bad.  The problem is the overall regression model is not significant.  He also has an interaction between baby's gender and outcome.   Is this possible?  Might it be a problem with multicollinearity?    Since it's a pilot study, should we be sticking with reporting partial correlations between pregnancy variables and baby's outcome variable - and partially out the variables we've entered on the first step of the regression?   Any ideas greatly appreciated!!  Many thanks for the help!   Susan
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 Was the first model with 4 variables significant? Dr. Paul R. Swank, Professor and Director of ResearchChildren's Learning InstituteUniversity of Texas Health Science Center-Houston From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of S CrawfordSent: Tuesday, March 29, 2011 12:51 PMTo: [hidden email]Subject: significant F change, but nonsignificant regression model overall Hi everyone,One of our post-docs is having troubles with a regression model he is trying to run.  He is trying to predict outcome in babies based on some pregnancy variables from the mothers collected during gestation.  He has 75 participants.  He has entered four variables to control for on the first step, and then two other predictors on the 2nd step.  So we're trying to see if these two predictors are significant above and beyond the four variables we are controlling for on the first step of the regression. The F change for adding these 2 predictors on Step 2 is significant - and the R2 change is .19 so not big, but not bad.  The problem is the overall regression model is not significant.  He also has an interaction between baby's gender and outcome. Is this possible?  Might it be a problem with multicollinearity?   Since it's a pilot study, should we be sticking with reporting partial correlations between pregnancy variables and baby's outcome variable - and partially out the variables we've entered on the first step of the regression? Any ideas greatly appreciated!!  Many thanks for the help! Susan
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by sgthomson99 Hi Susan,   Some things to try:   1. Assess the bivariate associations between all of your predictor variables to see if you have some strong correlations among your predictors. 2. Enter in one variable, the one you consider to be the most important, then enter in each variable by itself with this one variable and assess the change in the coefficient of your most important variable. This can give you an idea of the effect of one variable on another 2. Look at your tolerance to see if you have multicollinearity From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of S CrawfordSent: Tuesday, March 29, 2011 10:51 AMTo: [hidden email]Subject: significant F change, but nonsignificant regression model overall Hi everyone,One of our post-docs is having troubles with a regression model he is trying to run.  He is trying to predict outcome in babies based on some pregnancy variables from the mothers collected during gestation.  He has 75 participants.  He has entered four variables to control for on the first step, and then two other predictors on the 2nd step.  So we're trying to see if these two predictors are significant above and beyond the four variables we are controlling for on the first step of the regression. The F change for adding these 2 predictors on Step 2 is significant - and the R2 change is .19 so not big, but not bad.  The problem is the overall regression model is not significant.  He also has an interaction between baby's gender and outcome. Is this possible?  Might it be a problem with multicollinearity?   Since it's a pilot study, should we be sticking with reporting partial correlations between pregnancy variables and baby's outcome variable - and partially out the variables we've entered on the first step of the regression? Any ideas greatly appreciated!!  Many thanks for the help! Susan
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by sgthomson99 > Date: Tue, 29 Mar 2011 17:51:03 +0000 > From: [hidden email] > Subject: significant F change, but nonsignificant regression model overall > To: [hidden email] > > Hi everyone, > One of our post-docs is having troubles with a regression model he is > trying to run. He is trying to predict outcome in babies based on some > pregnancy variables from the mothers collected during gestation. He > has 75 participants. He has entered four variables to control for on > the first step, and then two other predictors on the 2nd step. So > we're trying to see if these two predictors are significant above and > beyond the four variables we are controlling for on the first step of > the regression. > > The F change for adding these 2 predictors on Step 2 is significant - > and the R2 change is .19 so not big, but not bad. The problem is the > overall regression model is not significant. He also has an > interaction between baby's gender and outcome. > > Is this possible? Might it be a problem with multicollinearity? > Assuming that the description is accurate, Paul Swank has pointed at the issue - the first four variables were not significant. But for a designed test of the two, their impact is thoroughly irrelevant. I repeat, the overall test of the regression is totally irrelevant, because the post-doc designated, at the start, the test of two variables. I am unsure of what you mean by saying "interaction" between gender and outcome.  "Outcome" usually means "what is being predicted". In a different style of testing, "interaction with outcome" denotes a main effect for one predictor (here: gender). In regression, "interaction" usually refers to a cross-product of two predictors, not Predictor and Outcome.  My tentative conclusion is that your statement about Gender is a summary of what shows up among the first 4 variables entered.  Should you take it seriously?  - Well, it was specified a-priori as a control variable.  I don't know what he wants to say about the 4 control variables, but that is a separate matter for discussion.  The designed test was positive. -- Rich Ulrich ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Rich Ulrich My apologies to everyone for mixing up the interaction.  I was in too much of a rush and definitely should have had our post-doc email the group with his questions.   The interaction is between baby's gender and one of the predictors.  For Model 1 with the 4 covariates entered, the Multiple R is .5, and F test is not significant. For Model 2 where he entered the 4 covariates on Step 1 and the 2 variables he is most interested in on Step 2, the Multiple R is .6 and F test is still not significant.But a priori, he was most interested in the 2 variables entered on Step 2 - and this is where the F change is significant.  One of the two variables is significant on Step 2. So can he focus on the F change being significant and ignore the fact that the overall model is not significant and ignore the fact that the 4 variables on Step 1 were not significant either?   Thanks so much.   Susan   > Date: Tue, 29 Mar 2011 14:29:26 -0400> From: [hidden email]> Subject: Re: significant F change, but nonsignificant regression model overall> To: [hidden email]> > > Date: Tue, 29 Mar 2011 17:51:03 +0000> > From: [hidden email]> > Subject: significant F change, but nonsignificant regression model overall> > To: [hidden email]> >> > Hi everyone,> > One of our post-docs is having troubles with a regression model he is> > trying to run. He is trying to predict outcome in babies based on some> > pregnancy variables from the mothers collected during gestation. He> > has 75 participants. He has entered four variables to control for on> > the first step, and then two other predictors on the 2nd step. So> > we're trying to see if these two predictors are significant above and> > beyond the four variables we are controlling for on the first step of> > the regression.> >> > The F change for adding these 2 predictors on Step 2 is significant -> > and the R2 change is .19 so not big, but not bad. The problem is the> > overall regression model is not significant. He also has an> > interaction between baby's gender and outcome.> >> > Is this possible? Might it be a problem with multicollinearity?> >> > Assuming that the description is accurate, Paul Swank has> pointed at the issue - the first four variables were not significant.> > But for a designed test of the two, their impact is thoroughly irrelevant.> I repeat, the overall test of the regression is totally irrelevant,> because the post-doc designated, at the start, the test of two variables.> > > I am unsure of what you mean by saying "interaction" between gender> and outcome. "Outcome" usually means "what is being predicted".> In a different style of testing, "interaction with outcome" denotes> a main effect for one predictor (here: gender).> > In regression, "interaction" usually refers to a cross-product of> two predictors, not Predictor and Outcome. My tentative conclusion> is that your statement about Gender is a summary of what shows up> among the first 4 variables entered. Should you take it seriously?> - Well, it was specified a-priori as a control variable. I don't> know what he wants to say about the 4 control variables, but that is> a separate matter for discussion. The designed test was positive.> > --> Rich Ulrich> > > > > =====================> To manage your subscription to SPSSX-L, send a message to> [hidden email] (not to SPSSX-L), with no body text except the> command. To leave the list, send the command> SIGNOFF SPSSX-L> For a list of commands to manage subscriptions, send the command> INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

Open this post in threaded view
|
Report Content as Inappropriate

Automatic reply: significant F change, but nonsignificant regression model overall

 I will be out of office on March 29 afternoon from 1pm.  I will have very limited access to email.   If you need immediate assistance please contact 479-575-2905.  Thank you.
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Mike Mike, You seem to have missed the comment,           He has entered four variables to control for on > > > the first step, and then two other predictors on the 2nd step. So > > > we're trying to see if these two predictors are significant above and > > > beyond the four variables we are controlling for on the first step of > > > the regression. ________________________________ > Date: Tue, 29 Mar 2011 15:51:48 -0400 > From: [hidden email] > Subject: Re: significant F change, but nonsignificant regression model > overall > To: [hidden email] > > Before making a serious recommendation, I think that I would like > to know more about the data. However, I would be hesitant about > reporting the "significant" change in F in the context of nonsignificant > models. Consider the following analog: a person conducts a one-way > ANOVA with six levels. The overall ANOVA is not significant but > post hoc testing reveals one contrast between means to be significant. [snip, rest of irrelevant example, and more] It's a designed test, so the other results are irrelevant. -- Rich Ulrich ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Swank, Paul R Hi Susan,   If this is a pilot study, can I say that you will have more data in the actual study? If profiling is necessary and you would like to avoid interaction, you might want to try C5 or other decision trees that profile with a dependent variable.   This might not be the best way but do note that data mining is data driven and you might require at least 300 data so that the data mining model could identify patterns in the data file.   Warmest Regards Dorraj Oet  Date: Tue, 29 Mar 2011 12:58:22 -0500From: [hidden email]Subject: Re: significant F change, but nonsignificant regression model overallTo: [hidden email] Was the first model with 4 variables significant?   Dr. Paul R. Swank, Professor and Director of Research Children's Learning Institute University of Texas Health Science Center-Houston   From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of S CrawfordSent: Tuesday, March 29, 2011 12:51 PMTo: [hidden email]Subject: significant F change, but nonsignificant regression model overall   Hi everyone,One of our post-docs is having troubles with a regression model he is trying to run.  He is trying to predict outcome in babies based on some pregnancy variables from the mothers collected during gestation.  He has 75 participants.  He has entered four variables to control for on the first step, and then two other predictors on the 2nd step.  So we're trying to see if these two predictors are significant above and beyond the four variables we are controlling for on the first step of the regression. The F change for adding these 2 predictors on Step 2 is significant - and the R2 change is .19 so not big, but not bad.  The problem is the overall regression model is not significant.  He also has an interaction between baby's gender and outcome. Is this possible?  Might it be a problem with multicollinearity?   Since it's a pilot study, should we be sticking with reporting partial correlations between pregnancy variables and baby's outcome variable - and partially out the variables we've entered on the first step of the regression? Any ideas greatly appreciated!!  Many thanks for the help! Susan
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Rich Ulrich On Tuesday, March 29, 2011 11:36 pm, Rich Ulrich wrote: > > Mike, > You seem to have missed the comment, > >>>> He has entered four variables to control for on >>>> the first step, and then two other predictors on the 2nd step. So >>>> we're trying to see if these two predictors are significant above and >>>> beyond the four variables we are controlling for on the first step of >>>> the regression. No, I didn't miss this comment.  Let's review what we might know about the situation (at least from my perspective): (1) The analyst is doing setwise regression, comparable to an ANCOVA, entering 4 variables/covariates as the first set.  As mentioned elsewhere, these covariates are NOT significantly related to the dependent variable. This implies that the multiple correlation and its squared version are zero, or R1=0.00.  One could, I think, legitimately ask why did one continue to use these as covariates or keep them in the model when the second set was entered -- one argument could be based on the expectation that there is a supressor relationship among the predictors but until we hear from the person who actually ran the analysis, I don't believe this was the strategy. (2) After the second set of predictors were entered there still was NO significant relationship between the predictors and the dependent variable. So, for this model R and R^2 are both equal to zero or R2=0.00 (3) There is a "significant increase in R^2" (F change) when the second set of predictors was entered.  This has me puzzled.  It is not clear to me why or how this could occur.  If R1(set 1/model 1)=0.00 and R2(set 2/model 2)=0.00, then why would R2-R1 != 0.00?  I suspect that maybe there really is a pattern of relationships present but that there is insufficient statistical power to detect them (the researcher either needs to get more subjects or better measurements). There may be other reasons but I think one needs to examine the data in order to figure out (one explanation is that it is just a Type I error). Rich, how would you explain what happens in (3) above? -Mike Palij New York University [hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 The problem is one of sample size. The original control variables do not result in a significant model. Does this mean they have no effect? No, it means you don't have enough power to detect that size effect. It may be that the effect size is worrisome enough to demand control even in the absence of significance. If the question really is do x5 and x6 really predict over and above x1 through x4, then they should probably be included. However, if x5 and x6 add significantly to x1-x4, then the fact that x1- x4 do not account for significant variability can pull down the R squared for the full model. Given this is a pilot study, I think we might be okay saying that x5 and x6 do predict significantly over and above x1 - x4 but I would look more at effect size here than statistical significance. The pilot study should be to estimate effect size so that we can design a suitably large full scale study to actually test the hypothesis. Dr. Paul R. Swank, Professor and Director of Research Children's Learning Institute University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Michael Palij Sent: Wednesday, March 30, 2011 6:34 AM To: [hidden email] Subject: Re: significant F change, but nonsignificant regression model overall On Tuesday, March 29, 2011 11:36 pm, Rich Ulrich wrote: > > Mike, > You seem to have missed the comment, > >>>> He has entered four variables to control for on >>>> the first step, and then two other predictors on the 2nd step. So >>>> we're trying to see if these two predictors are significant above and >>>> beyond the four variables we are controlling for on the first step of >>>> the regression. No, I didn't miss this comment.  Let's review what we might know about the situation (at least from my perspective): (1) The analyst is doing setwise regression, comparable to an ANCOVA, entering 4 variables/covariates as the first set.  As mentioned elsewhere, these covariates are NOT significantly related to the dependent variable. This implies that the multiple correlation and its squared version are zero, or R1=0.00.  One could, I think, legitimately ask why did one continue to use these as covariates or keep them in the model when the second set was entered -- one argument could be based on the expectation that there is a supressor relationship among the predictors but until we hear from the person who actually ran the analysis, I don't believe this was the strategy. (2) After the second set of predictors were entered there still was NO significant relationship between the predictors and the dependent variable. So, for this model R and R^2 are both equal to zero or R2=0.00 (3) There is a "significant increase in R^2" (F change) when the second set of predictors was entered.  This has me puzzled.  It is not clear to me why or how this could occur.  If R1(set 1/model 1)=0.00 and R2(set 2/model 2)=0.00, then why would R2-R1 != 0.00?  I suspect that maybe there really is a pattern of relationships present but that there is insufficient statistical power to detect them (the researcher either needs to get more subjects or better measurements). There may be other reasons but I think one needs to examine the data in order to figure out (one explanation is that it is just a Type I error). Rich, how would you explain what happens in (3) above? -Mike Palij New York University [hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 "Swank, Paul R" On Wednesday, March 30, 2011 11:06 AM, Paul Swank wrote: >The problem is one of sample size. The original control variables >do not result in a significant model. Does this mean they have no >effect? No, it means you don't have enough power to detect that >size effect. A null result implies two possible conditions: (1)  The null hypothesis is true. (2)  The null hypothesis is false but there is insufficient power to reject it. Which one of the above conditions one chooses to believe depends on a bunch of factors, such as previous research that is comparable to the current study -- if this is truly a pilot study where no one has done something like this before, then condition (1) is, I think, the more prudent choice.  However, given peoples' cognitive biases, including the sunk cost effect, one is probably loathe to entertain codnition (1) because no wants one research to support null results outside of SEM or other modeling situations. Of course, as you point out below, one way to determine which condition is most consistent with the evidence is to (a) define a specific effect size that one wants to detect, (b) specify a specific level of statistical power (say, between .80 to .95), and (c) then identify the sample size needed to detect the specified effect size. One might politely ask if this was done before the collection of this data, a good practice that is more often observed in the breach. Perhaps one should read Jack Cohen's writings before going to sleep at night to remember what good research conduct is.  If one did, then there would be less ambiguity about which of the two conditions above holds.  If one knows what effect size one wants to detect and one has appropriate power (say, .95-.99), then a null result is clearly more consistent with condition (1).  A retrospective power analysis is clearly indicated if one believes that condition (2) holds. -Mike Palij New York University [hidden email] >It may be that the effect size is worrisome enough to demand control >even in the absence of significance. If the question really is do x5 and >x6 really predict over and above x1 through x4, then they should >probably be included. However, if x5 and x6 add significantly to x1-x4, >then the fact that x1- x4 do not account for significant variability can pull >down the R squared for the full model. Given this is a pilot study, I think >we might be okay saying that x5 and x6 do predict significantly over >and above x1 - x4 but I would look more at effect size here than statistical >significance. The pilot study should be to estimate effect size so that we >can design a suitably large full scale study to actually test the hypothesis. > >Dr. Paul R. Swank, >Professor and Director of Research >Children's Learning Institute >University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Michael Palij Sent: Wednesday, March 30, 2011 6:34 AM To: [hidden email] Subject: Re: significant F change, but nonsignificant regression model overall On Tuesday, March 29, 2011 11:36 pm, Rich Ulrich wrote: > > Mike, > You seem to have missed the comment, > >>>> He has entered four variables to control for on >>>> the first step, and then two other predictors on the 2nd step. So >>>> we're trying to see if these two predictors are significant above and >>>> beyond the four variables we are controlling for on the first step of >>>> the regression. No, I didn't miss this comment.  Let's review what we might know about the situation (at least from my perspective): (1) The analyst is doing setwise regression, comparable to an ANCOVA, entering 4 variables/covariates as the first set.  As mentioned elsewhere, these covariates are NOT significantly related to the dependent variable. This implies that the multiple correlation and its squared version are zero, or R1=0.00.  One could, I think, legitimately ask why did one continue to use these as covariates or keep them in the model when the second set was entered -- one argument could be based on the expectation that there is a supressor relationship among the predictors but until we hear from the person who actually ran the analysis, I don't believe this was the strategy. (2) After the second set of predictors were entered there still was NO significant relationship between the predictors and the dependent variable. So, for this model R and R^2 are both equal to zero or R2=0.00 (3) There is a "significant increase in R^2" (F change) when the second set of predictors was entered.  This has me puzzled.  It is not clear to me why or how this could occur.  If R1(set 1/model 1)=0.00 and R2(set 2/model 2)=0.00, then why would R2-R1 != 0.00?  I suspect that maybe there really is a pattern of relationships present but that there is insufficient statistical power to detect them (the researcher either needs to get more subjects or better measurements). There may be other reasons but I think one needs to examine the data in order to figure out (one explanation is that it is just a Type I error). Rich, how would you explain what happens in (3) above? -Mike Palij New York University [hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 Actually the null is never true. Sometimes it is not very false. Dr. Paul R. Swank, Professor and Director of Research Children's Learning Institute University of Texas Health Science Center-Houston -----Original Message----- From: Mike Palij [mailto:[hidden email]] Sent: Wednesday, March 30, 2011 11:34 AM To: Swank, Paul R; [hidden email] Cc: Mike Palij Subject: Re: significant F change, but nonsignificant regression model overall "Swank, Paul R" On Wednesday, March 30, 2011 11:06 AM, Paul Swank wrote: >The problem is one of sample size. The original control variables >do not result in a significant model. Does this mean they have no >effect? No, it means you don't have enough power to detect that >size effect. A null result implies two possible conditions: (1)  The null hypothesis is true. (2)  The null hypothesis is false but there is insufficient power to reject it. Which one of the above conditions one chooses to believe depends on a bunch of factors, such as previous research that is comparable to the current study -- if this is truly a pilot study where no one has done something like this before, then condition (1) is, I think, the more prudent choice.  However, given peoples' cognitive biases, including the sunk cost effect, one is probably loathe to entertain codnition (1) because no wants one research to support null results outside of SEM or other modeling situations. Of course, as you point out below, one way to determine which condition is most consistent with the evidence is to (a) define a specific effect size that one wants to detect, (b) specify a specific level of statistical power (say, between .80 to .95), and (c) then identify the sample size needed to detect the specified effect size. One might politely ask if this was done before the collection of this data, a good practice that is more often observed in the breach. Perhaps one should read Jack Cohen's writings before going to sleep at night to remember what good research conduct is.  If one did, then there would be less ambiguity about which of the two conditions above holds.  If one knows what effect size one wants to detect and one has appropriate power (say, .95-.99), then a null result is clearly more consistent with condition (1).  A retrospective power analysis is clearly indicated if one believes that condition (2) holds. -Mike Palij New York University [hidden email] >It may be that the effect size is worrisome enough to demand control >even in the absence of significance. If the question really is do x5 and >x6 really predict over and above x1 through x4, then they should >probably be included. However, if x5 and x6 add significantly to x1-x4, >then the fact that x1- x4 do not account for significant variability can pull >down the R squared for the full model. Given this is a pilot study, I think >we might be okay saying that x5 and x6 do predict significantly over >and above x1 - x4 but I would look more at effect size here than statistical >significance. The pilot study should be to estimate effect size so that we >can design a suitably large full scale study to actually test the hypothesis. > >Dr. Paul R. Swank, >Professor and Director of Research >Children's Learning Institute >University of Texas Health Science Center-Houston -----Original Message----- From: SPSSX(r) Discussion [mailto:[hidden email]] On Behalf Of Michael Palij Sent: Wednesday, March 30, 2011 6:34 AM To: [hidden email] Subject: Re: significant F change, but nonsignificant regression model overall On Tuesday, March 29, 2011 11:36 pm, Rich Ulrich wrote: > > Mike, > You seem to have missed the comment, > >>>> He has entered four variables to control for on >>>> the first step, and then two other predictors on the 2nd step. So >>>> we're trying to see if these two predictors are significant above and >>>> beyond the four variables we are controlling for on the first step of >>>> the regression. No, I didn't miss this comment.  Let's review what we might know about the situation (at least from my perspective): (1) The analyst is doing setwise regression, comparable to an ANCOVA, entering 4 variables/covariates as the first set.  As mentioned elsewhere, these covariates are NOT significantly related to the dependent variable. This implies that the multiple correlation and its squared version are zero, or R1=0.00.  One could, I think, legitimately ask why did one continue to use these as covariates or keep them in the model when the second set was entered -- one argument could be based on the expectation that there is a supressor relationship among the predictors but until we hear from the person who actually ran the analysis, I don't believe this was the strategy. (2) After the second set of predictors were entered there still was NO significant relationship between the predictors and the dependent variable. So, for this model R and R^2 are both equal to zero or R2=0.00 (3) There is a "significant increase in R^2" (F change) when the second set of predictors was entered.  This has me puzzled.  It is not clear to me why or how this could occur.  If R1(set 1/model 1)=0.00 and R2(set 2/model 2)=0.00, then why would R2-R1 != 0.00?  I suspect that maybe there really is a pattern of relationships present but that there is insufficient statistical power to detect them (the researcher either needs to get more subjects or better measurements). There may be other reasons but I think one needs to examine the data in order to figure out (one explanation is that it is just a Type I error). Rich, how would you explain what happens in (3) above? -Mike Palij New York University [hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Mike > Date: Wed, 30 Mar 2011 07:34:20 -0400 > From: [hidden email] > Subject: Re: significant F change, but nonsignificant regression model overall > To: [hidden email] > > On Tuesday, March 29, 2011 11:36 pm, Rich Ulrich wrote: > > > > Mike, > > You seem to have missed the comment, > > > >>>> He has entered four variables to control for on > >>>> the first step, and then two other predictors on the 2nd step. So > >>>> we're trying to see if these two predictors are significant above and > >>>> beyond the four variables we are controlling for on the first step of > >>>> the regression. > > No, I didn't miss this comment. Let's review what we might know about > the situation (at least from my perspective): > > (1) The analyst is doing setwise regression, comparable to an ANCOVA, > entering 4 variables/covariates as the first set. As mentioned elsewhere, > these covariates are NOT significantly related to the dependent variable. Mike, No, they are not "doing setwise regression", whatever that new phrase means, if that is what you intended.  And they are certainly not doing Stepwise regression, which is what you seem to discuss later. The analysis used an intentional, pre-designated order of entry of terms. The statistical tool was a regression program.  There were 4 variables which were "controlled for", as explicitly described.  - That analysis tests two variable, with 4 variables being "controlled for." An analogous ANCOVA would be a two-factor design with  4 covariates, where the covariates are included for ... whatever purposes.  In more detail -- The "whole ANOVA"  will have a test with  d.f.= (4+groups-1).  The test (or tests) on the two factors do *not*  rely on the covariates being either significant or not-significant. If you control for something highly correlated with outcome (pre-scores, often), the covariates are highly significant.  If you control for "nuisance" variables, you hope that the nuisance variables do not have much effect because that complicates interpretations.  But in either case, you do *not*  use the overall test on (covariates + hypotheses) as a guide to the inference on hypotheses. [snip, description of a "stepwise" process; irrelevant to discussion of testing taken as a defined hierarchy.] -- Rich Ulrich ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

 In reply to this post by Rich Ulrich On Wednesday, March 30, 2011 2:55 PM, Rich Ulrich wrote: >Mike Palij wrote: >> No, I didn't miss this comment. Let's review what we might know about >> the situation (at least from my perspective): >> >> (1) The analyst is doing setwise regression, comparable to an ANCOVA, >> entering 4 variables/covariates as the first set. As mentioned elsewhere, >> these covariates are NOT significantly related to the dependent variable. > >Mike, >No, they are not "doing setwise regression", whatever that new >phrase means, if that is what you intended. That "new phrase" can be found in Cohen and Cohen (1975) in their Chapter 4 "Sets of Independent Variables". Of particular relevance is section 4.2 "The simultaneous and hierarchical models for sets". What you and the OP described was a hierarchical or sequential setwise regression analysis.  See pp127-144 if you have a copy handy.  If anything, you should say "whatever that arcane phrase means". As for your description of the analysis, do you really keep variables that don't provide any useful information in the equation?  I hope you report shrunken or adjusted R^2 when you report your results because they should be considerably smaller than R^2 as a result of the additional useless predictors.  It should give a person pause. -Mike Palij New York University [hidden email] ===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Open this post in threaded view
|
Report Content as Inappropriate

Re: significant F change, but nonsignificant regression model overall

Open this post in threaded view
|
Report Content as Inappropriate