bootstrap CI for dep t confusing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

bootstrap CI for dep t confusing

msherman

Hi All: I just ran a dependent t-test with bootstrapping. The non bootstrap result provides a t test value of 2.971 (10df) with a two tailed p value of .014. The bootstrap results from 1000 samples  provides a two-tailed p value of .072. So I understand that the bootstrapping might be in disagreement with the non bootstrap result. OK. However, when I look at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give the bootstrap p value of non-significance I was expected the 95% CI  to include 0. But it doesn’t.  Any ideas why I am getting such a result. Thanks,  martin

 

 

 

 

Martin F. Sherman, Ph.D.

Professor of Psychology

Director of Masters Education: Thesis Track

Loyola University Maryland

4501 North Charles Street

222 B Beatty Hall

Baltimore, MD 21210

 

410 617-2417

[hidden email]

 

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

bdates
Martin,

It's probably your small N. Since bootstrapping samples with replacement, it's very likely that you're getting multiple skewed samples, increasing the variance and therefore the CI.

Brian

________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of Martin Sherman [[hidden email]]
Sent: Monday, November 20, 2017 3:31 PM
To: [hidden email]
Subject: bootstrap CI for dep t confusing

Hi All: I just ran a dependent t-test with bootstrapping. The non bootstrap result provides a t test value of 2.971 (10df) with a two tailed p value of .014. The bootstrap results from 1000 samples  provides a two-tailed p value of .072. So I understand that the bootstrapping might be in disagreement with the non bootstrap result. OK. However, when I look at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give the bootstrap p value of non-significance I was expected the 95% CI  to include 0. But it doesn’t.  Any ideas why I am getting such a result. Thanks,  martin




Martin F. Sherman, Ph.D.
Professor of Psychology
Director of Masters Education: Thesis Track
Loyola University Maryland
4501 North Charles Street
222 B Beatty Hall
Baltimore, MD 21210

410 617-2417
[hidden email]

===================== To manage your subscription to SPSSX-L, send a message to [hidden email]<mailto:[hidden email]> (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

Mike
I may be confused but the problem is not that the variance has been increased
but that the CI is not wide enough to include 0.00.  If the variances have been increased,
then the standard error of the mean difference should have also been increased,
right?  The critical t-value for df=10 remains the same, so an increased
standard error would produce a wider CI if the formula used is
Mean difference +/- (crit t(10))*SE of Mean Difference

Or am I missing something?

-Mike Palij
New York University


On Mon, Nov 20, 2017 at 3:49 PM, Dates, Brian <[hidden email]> wrote:
Martin,

It's probably your small N. Since bootstrapping samples with replacement, it's very likely that you're getting multiple skewed samples, increasing the variance and therefore the CI.

Brian

________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of Martin Sherman [[hidden email]]
Sent: Monday, November 20, 2017 3:31 PM
To: [hidden email]
Subject: bootstrap CI for dep t confusing

Hi All: I just ran a dependent t-test with bootstrapping. The non bootstrap result provides a t test value of 2.971 (10df) with a two tailed p value of .014. The bootstrap results from 1000 samples  provides a two-tailed p value of .072. So I understand that the bootstrapping might be in disagreement with the non bootstrap result. OK. However, when I look at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give the bootstrap p value of non-significance I was expected the 95% CI  to include 0. But it doesn’t.  Any ideas why I am getting such a result. Thanks,  martin




Martin F. Sherman, Ph.D.
Professor of Psychology
Director of Masters Education: Thesis Track
Loyola University Maryland
4501 North Charles Street
222 B Beatty Hall
Baltimore, MD 21210

410 617-2417
[hidden email]

===================== To manage your subscription to SPSSX-L, send a message to [hidden email]<mailto:[hidden email]> (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

Bruce Weaver
Administrator
In reply to this post by msherman
I don't think it is uncommon for bootstrapped CIs and p-values to "disagree".
E.g.,

https://stats.stackexchange.com/questions/302928/disagreement-between-p-values-and-confidence-intervals

https://stats.stackexchange.com/questions/29546/p-value-vs-confidence-interval-obtained-in-bootstrapping

HTH.


msherman wrote

> Hi All: I just ran a dependent t-test with bootstrapping. The non
> bootstrap result provides a t test value of 2.971 (10df) with a two tailed
> p value of .014. The bootstrap results from 1000 samples  provides a
> two-tailed p value of .072. So I understand that the bootstrapping might
> be in disagreement with the non bootstrap result. OK. However, when I look
> at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give
> the bootstrap p value of non-significance I was expected the 95% CI  to
> include 0. But it doesn't.  Any ideas why I am getting such a result.
> Thanks,  martin
>
>
>
>
> Martin F. Sherman, Ph.D.
> Professor of Psychology
> Director of Masters Education: Thesis Track
> Loyola University Maryland
> 4501 North Charles Street
> 222 B Beatty Hall
> Baltimore, MD 21210
>
> 410 617-2417

> msherman@

>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD
--
Bruce Weaver
bweaver@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

Mike
Hmmm, after doing some review of literature, a couple of questions arise

(1)  The bootstrap CI is supposed to be provide a better estimate of the
CI (if one believe such things) than the normal theory CI one usually
calculates either with Z or t.  Situations where it appears that bootstrap CI
does worse or a situation as described by the OP, indicates some
pathological condition.

(2) One has to identify which bootstrap CI is being used because I reckon
that there are probably more than a half dozen versions. 

The following article by DiCiccio and Efron provides more information on
both of these points but there are recent articles that provide additional
perspectives:

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals.
Statistical science, 189-212.

This article is available on the Project Euclid website: click on:

It would probably help if the OP identified which bootstrap CI method
was used.  The methods are listed in the SPSS Algorithms manual.

-Mike Palij
New York University


On Mon, Nov 20, 2017 at 9:38 PM, Bruce Weaver <[hidden email]> wrote:
I don't think it is uncommon for bootstrapped CIs and p-values to "disagree".
E.g.,

https://stats.stackexchange.com/questions/302928/disagreement-between-p-values-and-confidence-intervals

https://stats.stackexchange.com/questions/29546/p-value-vs-confidence-interval-obtained-in-bootstrapping

HTH.


msherman wrote
> Hi All: I just ran a dependent t-test with bootstrapping. The non
> bootstrap result provides a t test value of 2.971 (10df) with a two tailed
> p value of .014. The bootstrap results from 1000 samples  provides a
> two-tailed p value of .072. So I understand that the bootstrapping might
> be in disagreement with the non bootstrap result. OK. However, when I look
> at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give
> the bootstrap p value of non-significance I was expected the 95% CI  to
> include 0. But it doesn't.  Any ideas why I am getting such a result.
> Thanks,  martin
>
>
>
>
> Martin F. Sherman, Ph.D.
> Professor of Psychology
> Director of Masters Education: Thesis Track
> Loyola University Maryland
> 4501 North Charles Street
> 222 B Beatty Hall
> Baltimore, MD 21210
>
> 410 617-2417

> msherman@

>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

Jon Peck
Based on the nice DiCiccio and Efron article, it seems that bootstrapping using method BCa, which is available but not the default in Statistics, would be superior for CI purposes.

On Tue, Nov 21, 2017 at 4:31 AM, Michael Palij <[hidden email]> wrote:
Hmmm, after doing some review of literature, a couple of questions arise

(1)  The bootstrap CI is supposed to be provide a better estimate of the
CI (if one believe such things) than the normal theory CI one usually
calculates either with Z or t.  Situations where it appears that bootstrap CI
does worse or a situation as described by the OP, indicates some
pathological condition.

(2) One has to identify which bootstrap CI is being used because I reckon
that there are probably more than a half dozen versions. 

The following article by DiCiccio and Efron provides more information on
both of these points but there are recent articles that provide additional
perspectives:

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals.
Statistical science, 189-212.

This article is available on the Project Euclid website: click on:

It would probably help if the OP identified which bootstrap CI method
was used.  The methods are listed in the SPSS Algorithms manual.

-Mike Palij
New York University


On Mon, Nov 20, 2017 at 9:38 PM, Bruce Weaver <[hidden email]> wrote:
I don't think it is uncommon for bootstrapped CIs and p-values to "disagree".
E.g.,

https://stats.stackexchange.com/questions/302928/disagreement-between-p-values-and-confidence-intervals

https://stats.stackexchange.com/questions/29546/p-value-vs-confidence-interval-obtained-in-bootstrapping

HTH.


msherman wrote
> Hi All: I just ran a dependent t-test with bootstrapping. The non
> bootstrap result provides a t test value of 2.971 (10df) with a two tailed
> p value of .014. The bootstrap results from 1000 samples  provides a
> two-tailed p value of .072. So I understand that the bootstrapping might
> be in disagreement with the non bootstrap result. OK. However, when I look
> at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give
> the bootstrap p value of non-significance I was expected the 95% CI  to
> include 0. But it doesn't.  Any ideas why I am getting such a result.
> Thanks,  martin
>
>
>
>
> Martin F. Sherman, Ph.D.
> Professor of Psychology
> Director of Masters Education: Thesis Track
> Loyola University Maryland
> 4501 North Charles Street
> 222 B Beatty Hall
> Baltimore, MD 21210
>
> <a href="tel:(410)%20617-2417" value="+14106172417" target="_blank">410 617-2417

> msherman@

>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD



--
Jon K Peck
[hidden email]

===================== To manage your subscription to SPSSX-L, send a message to [hidden email] (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD
Reply | Threaded
Open this post in threaded view
|

Re: bootstrap CI for dep t confusing

bdates
I have a paper with a complete list of resampling methods, including explanations, with formulae of 7 types of bootstrapping and the approximate bootstrap method, which is really a Bayesian approach to an actual bootstrap. It was the basis for Chapter 16 in The SAGE Handbook of Quantitative Methods in Psychology edited by Millsap and Maydeau-Olivares. It's very readable and a good reference for any resampling method, permutation, jackknife, and bootstrap. If you'd like a copy please indicate that. I believe that I can upload a file to Nabble if requested.

Brian

________________________________________
From: SPSSX(r) Discussion [[hidden email]] on behalf of Jon Peck [[hidden email]]
Sent: Tuesday, November 21, 2017 11:11 AM
To: [hidden email]
Subject: Re: bootstrap CI for dep t confusing

Based on the nice DiCiccio and Efron article, it seems that bootstrapping using method BCa, which is available but not the default in Statistics, would be superior for CI purposes.

On Tue, Nov 21, 2017 at 4:31 AM, Michael Palij <[hidden email]<mailto:[hidden email]>> wrote:
Hmmm, after doing some review of literature, a couple of questions arise

(1)  The bootstrap CI is supposed to be provide a better estimate of the
CI (if one believe such things) than the normal theory CI one usually
calculates either with Z or t.  Situations where it appears that bootstrap CI
does worse or a situation as described by the OP, indicates some
pathological condition.

(2) One has to identify which bootstrap CI is being used because I reckon
that there are probably more than a half dozen versions.

The following article by DiCiccio and Efron provides more information on
both of these points but there are recent articles that provide additional
perspectives:

DiCiccio, T. J., & Efron, B. (1996). Bootstrap confidence intervals.
Statistical science, 189-212.

This article is available on the Project Euclid website: click on:
http://projecteuclid.org/download/pdf_1/euclid.ss/1032280214

It would probably help if the OP identified which bootstrap CI method
was used.  The methods are listed in the SPSS Algorithms manual.

-Mike Palij
New York University
[hidden email]<mailto:[hidden email]>


On Mon, Nov 20, 2017 at 9:38 PM, Bruce Weaver <[hidden email]<mailto:[hidden email]>> wrote:
I don't think it is uncommon for bootstrapped CIs and p-values to "disagree".
E.g.,

https://stats.stackexchange.com/questions/302928/disagreement-between-p-values-and-confidence-intervals

https://stats.stackexchange.com/questions/29546/p-value-vs-confidence-interval-obtained-in-bootstrapping

HTH.


msherman wrote

> Hi All: I just ran a dependent t-test with bootstrapping. The non
> bootstrap result provides a t test value of 2.971 (10df) with a two tailed
> p value of .014. The bootstrap results from 1000 samples  provides a
> two-tailed p value of .072. So I understand that the bootstrapping might
> be in disagreement with the non bootstrap result. OK. However, when I look
> at the bootstrap 95% CI I find the interval is from 0.3636 to 1.9090. Give
> the bootstrap p value of non-significance I was expected the 95% CI  to
> include 0. But it doesn't.  Any ideas why I am getting such a result.
> Thanks,  martin
>
>
>
>
> Martin F. Sherman, Ph.D.
> Professor of Psychology
> Director of Masters Education: Thesis Track
> Loyola University Maryland
> 4501 North Charles Street<https://maps.google.com/?q=4501+North+Charles+Street&entry=gmail&source=g>
> 222 B Beatty Hall
> Baltimore, MD 21210
>
> 410 617-2417<tel:(410)%20617-2417>

> msherman@

>
>
> =====================
> To manage your subscription to SPSSX-L, send a message to

> LISTSERV@.UGA

>  (not to SPSSX-L), with no body text except the
> command. To leave the list, send the command
> SIGNOFF SPSSX-L
> For a list of commands to manage subscriptions, send the command
> INFO REFCARD





-----
--
Bruce Weaver
[hidden email]<mailto:[hidden email]>
http://sites.google.com/a/lakeheadu.ca/bweaver/

"When all else fails, RTFM."

NOTE: My Hotmail account is not monitored regularly.
To send me an e-mail, please use the address shown above.

--
Sent from: http://spssx-discussion.1045642.n5.nabble.com/

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email]<mailto:[hidden email]> (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD

===================== To manage your subscription to SPSSX-L, send a message to [hidden email]<mailto:[hidden email]> (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD



--
Jon K Peck
[hidden email]<mailto:[hidden email]>

===================== To manage your subscription to SPSSX-L, send a message to [hidden email]<mailto:[hidden email]> (not to SPSSX-L), with no body text except the command. To leave the list, send the command SIGNOFF SPSSX-L For a list of commands to manage subscriptions, send the command INFO REFCARD

=====================
To manage your subscription to SPSSX-L, send a message to
[hidden email] (not to SPSSX-L), with no body text except the
command. To leave the list, send the command
SIGNOFF SPSSX-L
For a list of commands to manage subscriptions, send the command
INFO REFCARD