Running a difference in differences test

10 messages
Open this post in threaded view
|

Running a difference in differences test

 I am trying to run a difference in differences test and I think my coding should be working but to the eye the results don't appear to be correct. I am talking about a difference in means of 1,000 compared to a difference in means of 100, which, again, seems it should be different. (It may be totally correct and the data is just got some crazy stuff going on, but I am a bit of a novice with this code and was hoping some more experienced people could give it a once over!) So I have data with 8 different conditions and am running with some covariates. I am testing here that condition 7-8 > 3-4. So I run a basic ANCOVA with contrast coding: UNIANOVA DV1 BY Condtion WITH CV1 CV2   /METHOD = SSTYPE(3) /LMATRIX 'DifDif' Condtion 0 0 1 -1 0 0 1 -1. Any help is appreciated. Thanks in advance!
Open this post in threaded view
|

Re: Running a difference in differences test

 Administrator I think you've got your signs the wrong way around on one pair of coefficients.  It helps if you include explicitly the contrasts for 7-8 and 3-4, and then compute the difference between them.   UNIANOVA DV1 BY Condtion WITH CV1 CV2   /METHOD = SSTYPE(3)   /LMATRIX '7-8'    Condtion 0 0  0  0 0 0 1 -1   /LMATRIX '3-4'    Condtion 0 0  1 -1 0 0 0  0   /LMATRIX 'DifDif' Condtion 0 0 -1  1 0 0 1 -1. HTH. UnderOver wrote I am trying to run a difference in differences test and I think my coding should be working but to the eye the results don't appear to be correct. I am talking about a difference in means of 1,000 compared to a difference in means of 100, which, again, seems it should be different. (It may be totally correct and the data is just got some crazy stuff going on, but I am a bit of a novice with this code and was hoping some more experienced people could give it a once over!) So I have data with 8 different conditions and am running with some covariates. I am testing here that condition 7-8 > 3-4. So I run a basic ANCOVA with contrast coding: UNIANOVA DV1 BY Condtion WITH CV1 CV2   /METHOD = SSTYPE(3) /LMATRIX 'DifDif' Condtion 0 0 1 -1 0 0 1 -1. Any help is appreciated. Thanks in advance! -- Bruce Weaver bweaver@lakeheadu.ca http://sites.google.com/a/lakeheadu.ca/bweaver/"When all else fails, RTFM." NOTE: My Hotmail account is not monitored regularly. To send me an e-mail, please use the address shown above.
Open this post in threaded view
|

Re: Running a difference in differences test

 In reply to this post by UnderOver Differences of differences often means crossed IVs. What are your 8 conditions?  Are they really 8 values of one construct or are they, for example, 3 dichotomous variables? Are all of the values between cases or are some of the values repeats? The substantive meaning of your variables and their values could well be informative to people who might help you. What is DV1? CV1? CV2? Art Kendall Social Research Consultants
Open this post in threaded view
|

Re: Running a difference in differences test

 Thanks for the response As for what these variables are the 8 conditions stem from 3 sets of dichotomous variables.   As for the DV it is a continuous variable that ranges from 0 to about 10,000 (dollars for investment). One CV is a categorical variable(wealth) and the other is continuous (Age).
Open this post in threaded view
|

Re: Running a difference in differences test

 In reply to this post by Bruce Weaver Thank you for the response. Yes you were right I had those backwards, but they are still giving me an odd response. Partly what prompted me to ask about this was that my first contrast (7-8) is shows a p-value >.05 while the second contrast is <.7 yet the difference in difference test is <.2, which just doesn't seem correct, although I realize is totally possible.
Open this post in threaded view
|

Re: Running a difference in differences test

 In reply to this post by UnderOver Then it appears that you want a 3 way ANCOVA. Whichever difference of differences you are interested in would be one of the interactions. It seems odd that wealth would be categorical.  How many values does it have?  Is it fully crossed with the three dichotomies? That would make a 4way ancova. How many cases do you have?  Are the cell sizes balanced? Art Kendall Social Research Consultants
Open this post in threaded view
|

Re: Running a difference in differences test

 Sorry I meant that it is ordinal, was going fast and mistakes abound. It has 4 values to match some regulatory items. I have run the 3-way ANCOVA and there are significant interactions with the IVs, so when digging into what is actual significant I have looked at the individual contrasts some are some aren't. Instead of having some no result hypotheses and tests I wanted to use this difference in difference test was a way to utilize the no result contrasts as part of the story showing where the contrast is significant vs. where it is not. Hopefully making for a more compelling study then a no results test would.
Open this post in threaded view
|

Re: Running a difference in differences test

 An interaction term is a difference of differences.  Did you hypothesize a particular difference of differences? When you do your visualization do you see crossed lines? Art Kendall Social Research Consultants